-- Logs begin at Sun 2022-10-16 00:10:34 UTC, end at Sun 2022-10-16 01:14:26 UTC. -- Oct 16 00:42:42.190461 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 16 00:42:42.203146 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 16 00:42:42.206450 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: *** Starting uWSGI 2.0.18-debian (64bit) on [Sun Oct 16 00:42:42 2022] *** Oct 16 00:42:42.206450 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 16 00:42:42.206450 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 16 00:42:42.206450 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: nodename: dv-cinder-master-3-iscsi Oct 16 00:42:42.206450 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: machine: x86_64 Oct 16 00:42:42.206450 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: clock source: unix Oct 16 00:42:42.206450 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: pcre jit disabled Oct 16 00:42:42.206450 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: detected number of CPU cores: 4 Oct 16 00:42:42.206450 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: current working directory: / Oct 16 00:42:42.207107 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: detected binary path: /usr/bin/uwsgi-core Oct 16 00:42:42.207107 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: your processes number limit is 47680 Oct 16 00:42:42.207107 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: your memory page size is 4096 bytes Oct 16 00:42:42.207107 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: detected max file descriptor number: 2048 Oct 16 00:42:42.207107 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: lock engine: pthread robust mutexes Oct 16 00:42:42.207107 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: thunder lock: enabled Oct 16 00:42:42.207107 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 16 00:42:42.207107 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 16 00:42:42.248157 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: Python main interpreter initialized at 0x561591af39c0 Oct 16 00:42:42.248157 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: python threads support enabled Oct 16 00:42:42.248157 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: your server socket listen backlog is limited to 100 connections Oct 16 00:42:42.248157 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: your mercy for graceful operations on workers is 90 seconds Oct 16 00:42:42.248572 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: mapped 403077 bytes (393 KB) for 2 cores Oct 16 00:42:42.248572 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: *** Operational MODE: preforking *** Oct 16 00:42:42.248881 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: *** uWSGI is running in multiple interpreter mode *** Oct 16 00:42:42.248881 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: spawned uWSGI master process (pid: 81472) Oct 16 00:42:42.249097 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 16 00:42:42.255217 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: spawned uWSGI worker 1 (pid: 81485, cores: 1) Oct 16 00:42:42.255217 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: spawned uWSGI worker 2 (pid: 81486, cores: 1) Oct 16 00:42:42.255217 dv-cinder-master-3-iscsi devstack@keystone.service[81472]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 16 00:42:46.646924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 16 00:42:46.646924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: from cryptography.utils import int_from_bytes Oct 16 00:42:46.646924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 16 00:42:46.646924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: from cryptography.utils import int_from_bytes Oct 16 00:42:46.647861 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 16 00:42:46.647861 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: from cryptography.utils import int_from_bytes Oct 16 00:42:46.647861 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 16 00:42:46.647861 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: from cryptography.utils import int_from_bytes Oct 16 00:42:46.692309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 16 00:42:46.692487 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 16 00:42:46.692619 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] command line args: [] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 16 00:42:46.692897 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 16 00:42:46.693080 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ================================================================================ {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 16 00:42:46.693346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] admin_token = **** {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.693598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] config_dir = [] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.693790 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.693978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] config_source = [] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.694162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] debug = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.694367 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.695557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.695557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.695557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.695557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.695795 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] list_limit = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.695867 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] log_config_append = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.696000 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.696180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] log_dir = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.696361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] log_file = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.696601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.696855 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.697108 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.697359 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.697540 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.697731 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.698005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.698204 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.698435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.698675 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.698932 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.699690 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.699827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 16 00:42:46.699996 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 16 00:42:46.700144 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.700262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] command line args: [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 16 00:42:46.700413 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 16 00:42:46.700536 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.700656 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ================================================================================ {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 16 00:42:46.700772 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.700886 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] admin_token = **** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.701171 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] config_dir = [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.701285 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.25/identity {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.701399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.701521 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] publish_errors = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.701634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] config_source = [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.701752 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.701865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] debug = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.701984 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.702106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.702249 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.702370 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.702482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.702593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.702717 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.702837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.703026 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] standard_threads = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.703143 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.703255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.703373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] list_limit = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.703495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.703599 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] log_config_append = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.703716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.703829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.703948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] use_journal = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.704075 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] log_dir = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.704465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] use_json = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.704586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] log_file = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.704699 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] use_stderr = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.704811 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.704929 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] use_syslog = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.705049 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.705173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.705294 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.705406 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.705526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.705647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.705760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.705884 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.705998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.706110 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.706238 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.706361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.706481 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.706603 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.706722 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.706722 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.706969 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.707096 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.707211 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.707324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.707442 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.707556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.707669 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.707834 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.707956 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.707956 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.708176 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.708289 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.708289 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.708520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.25/identity {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.708520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] publish_errors = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.708740 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.708853 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.708973 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.709087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.709200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.709312 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.709424 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.709535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.709656 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.709812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.709926 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.710038 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] standard_threads = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.710150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.710262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.710382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.710503 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.710503 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.710722 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.710841 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] use_journal = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.710977 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.711102 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] use_json = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.711217 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.711329 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] use_stderr = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.711329 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] use_syslog = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.711538 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.711650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 16 00:42:46.711773 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.711889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.712001 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.712120 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.712233 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.712356 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.712477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.712591 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.712703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.712703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.712921 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.713121 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.713245 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.713358 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.713470 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.713582 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.713694 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.713818 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.713934 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.714047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.714160 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.714278 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.714398 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.714523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.714638 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.714741 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.714936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.715052 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.715165 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.715294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.715409 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.715528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.715641 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.715753 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.715865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.715987 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.716107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.716227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.716341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.716453 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.716565 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.716687 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.716801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.716914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.717027 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.717146 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.717260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.717260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.717480 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.717593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.717593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.718040 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.718165 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.718291 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.718410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.718524 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.718635 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.718747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.718747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.719085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.719203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.719321 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.719434 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.719546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.719657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.719817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.719934 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.720046 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.720158 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.720276 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.720390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.720510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.720624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.720736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.720857 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.720970 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.721067 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.721172 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.721277 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.721379 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.721474 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.721568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.721668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.721777 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.721880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.721975 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.721975 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.722213 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.722323 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.722448 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.722563 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.722673 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.722793 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.722919 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.723045 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.723168 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.723283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.723397 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.723516 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.723631 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.723752 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.723899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.724025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.724140 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.724253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.724365 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.724479 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.724592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.724713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.724829 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.724942 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.725061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.725176 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.725289 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.725402 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.725516 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.725630 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.725742 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.725862 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.726014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.726141 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.726256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.726370 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.726485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.726597 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.726719 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.726836 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.727003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.727117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.727232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.727346 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.727460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.727587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.727702 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.727822 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.727939 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.727939 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.728151 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.728273 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.728388 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.728502 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.728616 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.728729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.728842 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.728966 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.729082 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.729195 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.729308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.729421 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.729534 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.729656 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.729772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.729917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.730040 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.730161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.730276 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.730998 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.731125 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.731246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.731362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.731475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.731598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.731723 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] auth.external = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.731846 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.731962 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.732076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.732076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.732076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.732398 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.732526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.732642 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.733061 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.733183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] auth.password = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.733297 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.733411 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] auth.token = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.733534 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.733650 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.733764 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.733879 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.733992 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.734112 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.734239 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.734355 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.734468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.734594 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.734715 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.734831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.735017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.735144 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.735268 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.735385 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] credential.caching = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.735506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.735622 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.735736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.735851 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.735975 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.736091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.736205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.736357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.736474 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.736597 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.736725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.736842 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.736957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.737072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.737186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.737306 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.737430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.737546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.737660 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.737774 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.737896 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.738014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.738140 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] auth.external = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.738257 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.738370 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.738469 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.738581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.738696 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.738818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.738948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.739064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] auth.password = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.739177 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] federation.caching = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.739291 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] auth.token = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.739406 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.739533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.739650 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.739764 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.739879 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.740028 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.740147 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.740271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.740387 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.740503 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.740624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.740741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.740857 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.740979 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.741096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.741212 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] credential.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.741326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.741441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.741555 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.741678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.741801 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.741917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.742032 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] identity.caching = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.742147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.742270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.742390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.742506 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.742621 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.742735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.742849 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.742996 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.743113 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.743228 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.743344 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.743459 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.743573 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.743695 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.743801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.743914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.744029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.744150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.744267 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.744390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.744518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.744643 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.744758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.744873 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.744993 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] federation.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.745180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.745298 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.745414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.745529 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.745651 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.745767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.745890 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.746078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.746200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.746316 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.746433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.746547 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.746672 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.746788 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.746945 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.747062 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.747178 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.747293 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.747421 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.747539 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.747654 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.747769 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.747769 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.747985 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.748107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.748224 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.748339 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.748453 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.748569 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.748683 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.748806 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.748929 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.749051 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.749168 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.749288 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.749404 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.749527 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.749644 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.749759 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.749873 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.749987 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.750135 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.750261 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.750378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.750494 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.750610 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.751182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.751309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.751440 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.751558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.751675 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.751791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.751906 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.752022 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.752146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.752263 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.752379 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.752379 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.752606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.752732 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.752849 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.752964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.753079 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.753194 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.753309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.753431 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.753548 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.753663 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.753779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.753893 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.754008 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.754131 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.754254 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.754371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.754486 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.754601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.754716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.754840 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.754972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.755088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.755203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.755317 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.755432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.755559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.755676 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.755791 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.755907 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.756022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.756136 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.756259 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.756376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.756492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.756607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.756721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.756835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.756960 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.757076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.757190 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.757305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.757420 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.757589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.757717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.757834 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.757950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.758064 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.758178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.758292 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.758414 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.758531 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.758651 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.758774 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.758993 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.759113 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.759240 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.759357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.759472 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.759587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.759702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.759817 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.759940 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.760057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.760206 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.760323 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.760438 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.760553 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.760675 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.760793 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.760908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.761023 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.761138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.761253 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.761375 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.761492 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.762128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.762251 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.762251 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.762466 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.762589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.762706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.762820 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.762951 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.763067 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.763184 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.763313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.763429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.763429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.763644 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.763759 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.763874 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.763998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.764114 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.764231 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.764346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] resource.caching = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.764462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.764577 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.764700 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.764817 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.764932 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.765047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.765162 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.765278 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.765401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.765518 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.765633 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.765633 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.765848 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.765962 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.766086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.766203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.766319 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.766434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.766556 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.766672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.766797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] role.caching = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.766964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.767083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] role.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.767083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.767309 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.767309 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.767535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.767652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.767767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.767889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.768006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.768130 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.768247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.768361 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.768477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.768592 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.768707 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.768896 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.769015 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.769129 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.769244 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.769360 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.769475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.769598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] resource.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.769715 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.769831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.769946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.770060 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.770175 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.770337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.770455 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.770572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.770687 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.770801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.770929 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.771057 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.771174 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.771290 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.771405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.771519 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.771519 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.771744 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.771860 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] role.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.771975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.772089 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] role.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.772204 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.772318 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.772441 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.772558 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.772674 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.772788 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.772908 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.772908 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.773135 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.773253 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.773367 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.773482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.773596 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.773713 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.773836 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.773953 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.774068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.774183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.774297 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.774420 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.774536 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.774651 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.774766 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.774939 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.775056 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.775183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] token.caching = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.775300 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.775416 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.775530 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.775694 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.775694 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.775919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.775919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.776136 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.776136 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.776350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.776465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.776588 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.776704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.776820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.776935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.777049 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.777222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.777419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.777537 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.777652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.777790 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.777944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.778060 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.778184 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.778308 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.778425 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.778531 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.778644 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.778759 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.778935 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.779069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.779187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.779187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.779403 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.779517 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.779632 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.connection = **** {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.779755 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.779871 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.779987 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.780102 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.780216 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.780362 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.780488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] token.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.780605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.780720 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.780835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.780951 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.781067 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.781190 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.781308 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.781423 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.781539 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.781654 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.781769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.781892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.782010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.782126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.782241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.782357 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.782472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.782595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.782712 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.782827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.782992 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.783109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.783225 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.783352 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.783469 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.783585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.783700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.783815 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.783931 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.784055 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.784182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81486) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 16 00:42:46.784311 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.784311 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.784311 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.connection = **** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.784311 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.784311 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.784311 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.784311 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.785039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.785039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.785039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.785354 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.785354 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.785576 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.785827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.786070 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.786324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.786566 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.786816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.787078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.787326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.787567 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 16 00:42:46.787741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81485) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 16 00:42:46.813189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:46.813422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:46.813652 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:46.814052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:46.814292 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:46.814525 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:46.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:46.822934 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:46.851559 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:46.851792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:46.852274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:46.852513 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:46.906609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:46.907069 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:46.957300 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:46.957484 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:46.959357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:46.959599 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.115336 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.115574 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.115795 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.116021 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.116164 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.116254 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.116401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.116483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.116630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.116854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.117077 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.117296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.140356 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.140586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.140816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.140918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.141042 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.141125 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.141266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.141349 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.141486 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.141570 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.141709 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.141789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.141929 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.142146 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.153955 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.154069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.166779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.166925 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.167024 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.167138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.167257 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.167366 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.167489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.167594 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.167716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.167800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.168046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.168168 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.168339 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81486) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.168431 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81485) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 16 00:42:47.185293 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.185510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 16 00:42:47.185798 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.185882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.186014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 16 00:42:47.186114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 16 00:42:47.186322 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.186403 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.186550 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 16 00:42:47.186631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.186831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.186939 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.187121 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.187201 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.187385 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.187647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.196788 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.197078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.197276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.197360 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.197566 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.197664 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.197839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.198125 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.205473 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.206321 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.208479 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.208726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.209018 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.209272 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.209353 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.209564 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.209661 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.209853 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.209935 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.210130 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.210212 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.210413 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.210496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.210680 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.210760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.210984 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.211080 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.211266 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.211347 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.211541 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.211820 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.212097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.235509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.236055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.237010 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.237305 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.237545 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.237839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.242221 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.242514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.242598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.242805 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.242918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.243119 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.249510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.249807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.249891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.250119 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.255707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.255986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.256092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.256293 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.256374 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.256588 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.256669 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.256878 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.256960 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.257159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.257249 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.257442 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.257522 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.257718 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.257797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.257995 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.273995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.274281 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.274364 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.274648 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.278295 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.278837 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.281580 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.281891 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.282075 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.282379 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.286162 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.286579 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.289363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.289687 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.292643 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.293440 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.293535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.293781 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.293932 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.294050 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.294343 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.294426 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.294637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.294718 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.294948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.295237 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.310576 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.310886 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.311170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.311279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.311441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.311577 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.311849 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.312113 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.318323 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.319021 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.319895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.320556 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.321471 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.321738 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.322092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.322374 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.324974 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.325268 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.325556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.325854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.325936 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.326226 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.326320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.326509 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.326793 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.327203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.337828 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.338116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.338400 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.338688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.338789 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.338999 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.339106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.339287 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.339398 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.339574 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.339878 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.340094 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.340545 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.340839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.355901 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.357103 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.358819 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.360056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.361876 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.362981 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.363414 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.364436 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.364898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.365199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.365614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.365902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.366317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.366736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.372110 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.373583 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.373666 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.375084 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.376532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.376939 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.377237 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.377525 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.377966 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.378368 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.378659 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.379186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.385382 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.385684 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.385974 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.387799 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.388286 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.388697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.393450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81485) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.393746 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.394035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.394329 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.394611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.394910 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.395198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.395484 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.395670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81486) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 16 00:42:47.395772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.395960 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.396041 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.396247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.396375 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.396530 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.396649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.396809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.396931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.397087 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.397207 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81485) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.397366 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.397656 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.397931 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.398228 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.398506 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.400996 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.400996 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.400996 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81486) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 16 00:42:47.432084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x561591af39c0 pid: 81485 (default app) Oct 16 00:42:47.435669 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x561591af39c0 pid: 81486 (default app) Oct 16 00:42:47.435810 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e53e6e-9ce6-4092-9758-0969c9455fdd None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:42:47.436022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e53e6e-9ce6-4092-9758-0969c9455fdd None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:42:47.436302 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e53e6e-9ce6-4092-9758-0969c9455fdd None None] PATH_INFO: `/v3` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:42:47.440588 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1/1] 10.222.0.25 () {50 vars in 791 bytes} [Sun Oct 16 00:42:47 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:42:54.266302 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8307b67f-46a3-40f0-b9a6-40ecac5e4a58 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:42:54.266851 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8307b67f-46a3-40f0-b9a6-40ecac5e4a58 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:42:54.266851 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8307b67f-46a3-40f0-b9a6-40ecac5e4a58 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:42:54.271078 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1/2] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:42:54 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:42:54.276748 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7928cf76-dc4f-4e2e-8cbc-f28b003e27d0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:42:54.277277 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7928cf76-dc4f-4e2e-8cbc-f28b003e27d0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:42:54.277379 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7928cf76-dc4f-4e2e-8cbc-f28b003e27d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:42:54.349403 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: INFO dbcounter [None req-7928cf76-dc4f-4e2e-8cbc-f28b003e27d0 None None] Registered counter for database keystone Oct 16 00:42:54.368564 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG oslo_db.sqlalchemy.engines [None req-7928cf76-dc4f-4e2e-8cbc-f28b003e27d0 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81485) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 16 00:42:54.375468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writer thread running {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 16 00:42:54.555084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG passlib.handlers.bcrypt [None req-7928cf76-dc4f-4e2e-8cbc-f28b003e27d0 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81485) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 16 00:42:54.555488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG passlib.handlers.bcrypt [None req-7928cf76-dc4f-4e2e-8cbc-f28b003e27d0 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81485) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 16 00:42:54.613774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-7928cf76-dc4f-4e2e-8cbc-f28b003e27d0 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:42:54.615415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7928cf76-dc4f-4e2e-8cbc-f28b003e27d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:42:54.743973 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 2/3] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:42:54 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 468 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 16 00:42:55.189448 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8307b67f-46a3-40f0-b9a6-40ecac5e4a58 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:42:55.260006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: INFO dbcounter [None req-8307b67f-46a3-40f0-b9a6-40ecac5e4a58 None None] Registered counter for database keystone Oct 16 00:42:55.282214 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG oslo_db.sqlalchemy.engines [None req-8307b67f-46a3-40f0-b9a6-40ecac5e4a58 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81486) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 16 00:42:55.288883 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writer thread running {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 16 00:42:55.410170 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8307b67f-46a3-40f0-b9a6-40ecac5e4a58 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:42:55.414591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f97fa5-a282-4b2c-9610-8727179c3619 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:42:55.415256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f97fa5-a282-4b2c-9610-8727179c3619 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:42:55.415542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f97fa5-a282-4b2c-9610-8727179c3619 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:42:55.415816 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f97fa5-a282-4b2c-9610-8727179c3619 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:42:55.425052 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f97fa5-a282-4b2c-9610-8727179c3619 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:42:55.477810 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: WARNING keystone.common.rbac_enforcer.enforcer [None req-47f97fa5-a282-4b2c-9610-8727179c3619 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 16 00:42:55.478085 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f97fa5-a282-4b2c-9610-8727179c3619 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:42:55.488504 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 2/4] 10.222.0.25 () {58 vars in 1324 bytes} [Sun Oct 16 00:42:55 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 16 00:42:55.494512 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7928cf76-dc4f-4e2e-8cbc-f28b003e27d0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:42:55.511394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7928cf76-dc4f-4e2e-8cbc-f28b003e27d0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:42:55.516357 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62507a04-2342-4272-b342-c9afd2a0f618 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:42:55.517477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62507a04-2342-4272-b342-c9afd2a0f618 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:42:55.517760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62507a04-2342-4272-b342-c9afd2a0f618 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:42:55.518040 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62507a04-2342-4272-b342-c9afd2a0f618 admin admin] PATH_INFO: `/v3/projects/f1d89d5c90cd4c35b26fbcfc82450844` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:42:55.519665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62507a04-2342-4272-b342-c9afd2a0f618 admin admin] RBAC: Authorizing `identity:get_project(project_id=f1d89d5c90cd4c35b26fbcfc82450844)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:42:55.572520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: WARNING keystone.common.rbac_enforcer.enforcer [None req-62507a04-2342-4272-b342-c9afd2a0f618 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 16 00:42:55.572873 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62507a04-2342-4272-b342-c9afd2a0f618 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:42:55.578108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 3/5] 10.222.0.25 () {56 vars in 1207 bytes} [Sun Oct 16 00:42:55 2022] GET /identity/v3/projects/f1d89d5c90cd4c35b26fbcfc82450844 => generated 344 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:42:57.751008 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab41fe1-c2bc-4628-896f-4663dadfb651 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:42:57.751540 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab41fe1-c2bc-4628-896f-4663dadfb651 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:42:57.751540 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab41fe1-c2bc-4628-896f-4663dadfb651 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:42:57.755633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 3/6] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:42:57 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:42:57.761061 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162a67d8-224e-4ca5-8fb1-06d33f6bf14e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:42:57.761963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162a67d8-224e-4ca5-8fb1-06d33f6bf14e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:42:57.762199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162a67d8-224e-4ca5-8fb1-06d33f6bf14e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:42:57.784079 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-162a67d8-224e-4ca5-8fb1-06d33f6bf14e None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:42:57.785651 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-162a67d8-224e-4ca5-8fb1-06d33f6bf14e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:42:57.793700 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 4/7] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:42:57 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 16 00:42:58.067780 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab41fe1-c2bc-4628-896f-4663dadfb651 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:42:58.080526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab41fe1-c2bc-4628-896f-4663dadfb651 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:42:58.084749 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b40b72f-511c-4604-b690-f28a8032aceb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:42:58.085351 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b40b72f-511c-4604-b690-f28a8032aceb admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:42:58.085637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b40b72f-511c-4604-b690-f28a8032aceb admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:42:58.085911 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b40b72f-511c-4604-b690-f28a8032aceb admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:42:58.094267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b40b72f-511c-4604-b690-f28a8032aceb admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:42:58.096082 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b40b72f-511c-4604-b690-f28a8032aceb admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:42:58.104808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 4/8] 10.222.0.25 () {58 vars in 1324 bytes} [Sun Oct 16 00:42:58 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 16 00:42:58.111208 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162a67d8-224e-4ca5-8fb1-06d33f6bf14e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:42:58.120331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162a67d8-224e-4ca5-8fb1-06d33f6bf14e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:42:58.124755 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e8b308-93b3-4f16-ac66-de78b5b6bd9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:42:58.125394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e8b308-93b3-4f16-ac66-de78b5b6bd9a admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:42:58.125681 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e8b308-93b3-4f16-ac66-de78b5b6bd9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:42:58.125960 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e8b308-93b3-4f16-ac66-de78b5b6bd9a admin admin] PATH_INFO: `/v3/users/68c5eeb505a240daa471059d6635deea` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:42:58.127316 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e8b308-93b3-4f16-ac66-de78b5b6bd9a admin admin] RBAC: Authorizing `identity:get_user(user_id=68c5eeb505a240daa471059d6635deea)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:42:58.129175 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e8b308-93b3-4f16-ac66-de78b5b6bd9a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:42:58.130731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 5/9] 10.222.0.25 () {56 vars in 1198 bytes} [Sun Oct 16 00:42:58 2022] GET /identity/v3/users/68c5eeb505a240daa471059d6635deea => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:00.358433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb57d813-eb81-42cf-9f80-c94aa2fb0e08 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:00.359006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb57d813-eb81-42cf-9f80-c94aa2fb0e08 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:00.359113 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb57d813-eb81-42cf-9f80-c94aa2fb0e08 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:00.360416 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 5/10] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:43:00.365761 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8790cb-3470-4e29-8244-f7a93c6a0553 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:00.366197 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8790cb-3470-4e29-8244-f7a93c6a0553 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:00.366400 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8790cb-3470-4e29-8244-f7a93c6a0553 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:00.384928 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-bf8790cb-3470-4e29-8244-f7a93c6a0553 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:00.386117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bf8790cb-3470-4e29-8244-f7a93c6a0553 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:00.396479 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d13024-c5a0-4f61-9861-6a197567169d None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:00.396767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d13024-c5a0-4f61-9861-6a197567169d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:00.397050 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d13024-c5a0-4f61-9861-6a197567169d None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:00.398404 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 6/11] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:43:00.404612 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b42eb6-44dc-44c1-bf2b-bbebab43fccc None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:00.404901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b42eb6-44dc-44c1-bf2b-bbebab43fccc None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:00.405188 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b42eb6-44dc-44c1-bf2b-bbebab43fccc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:00.448162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG passlib.handlers.bcrypt [None req-42b42eb6-44dc-44c1-bf2b-bbebab43fccc None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81486) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 16 00:43:00.448562 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG passlib.handlers.bcrypt [None req-42b42eb6-44dc-44c1-bf2b-bbebab43fccc None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81486) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 16 00:43:00.475149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 6/12] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:00.498483 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-42b42eb6-44dc-44c1-bf2b-bbebab43fccc None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:00.500160 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-42b42eb6-44dc-44c1-bf2b-bbebab43fccc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:00.602716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 7/13] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:00.744213 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8790cb-3470-4e29-8244-f7a93c6a0553 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:00.769258 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:00.853458 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8790cb-3470-4e29-8244-f7a93c6a0553 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:00.881293 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b42eb6-44dc-44c1-bf2b-bbebab43fccc None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:00.890020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e621a9a2-d6e5-467f-a0f9-7edeb430e001 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:00.890674 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e621a9a2-d6e5-467f-a0f9-7edeb430e001 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:00.891004 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e621a9a2-d6e5-467f-a0f9-7edeb430e001 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:00.891289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e621a9a2-d6e5-467f-a0f9-7edeb430e001 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:00.902998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e621a9a2-d6e5-467f-a0f9-7edeb430e001 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:00.912044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-e621a9a2-d6e5-467f-a0f9-7edeb430e001 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:43:00.912044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:00.912044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:00.912044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:00.912044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:00.912044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:00.912044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:00.912044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:00.912044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:00.912044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:00.912044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:00.912044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:00.912044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:00.912044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:00.912044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:00.912044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:00.913074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:00.913074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:00.913074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:00.913074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:00.913074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:00.913074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:00.913074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:00.913074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:00.913074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:00.913074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:00.913074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:00.913074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:00.913074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:00.913074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:00.913074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:00.913074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:00.914128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:00.914128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:00.914128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:00.914128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:00.914128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:00.914128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:00.914128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:00.914128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:00.914128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:00.914128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:00.914128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:43:00.914128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:00.914128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 7/14] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:43:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:00.919769 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e621a9a2-d6e5-467f-a0f9-7edeb430e001 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:00.988618 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b42eb6-44dc-44c1-bf2b-bbebab43fccc None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:00.991072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:01.010159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:01.030429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e621a9a2-d6e5-467f-a0f9-7edeb430e001 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:01.033134 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9ecb51-efc9-421f-a334-2cee610320ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:01.034016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9ecb51-efc9-421f-a334-2cee610320ed None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:01.034297 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9ecb51-efc9-421f-a334-2cee610320ed None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:01.034568 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9ecb51-efc9-421f-a334-2cee610320ed None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:01.035598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9ecb51-efc9-421f-a334-2cee610320ed None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:01.037037 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9ecb51-efc9-421f-a334-2cee610320ed None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:01.049870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.sql.core [None req-2b9ecb51-efc9-421f-a334-2cee610320ed None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 16 00:43:01.049870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 16 00:43:01.049870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [parameters: {'id': 'd2290b771f4348cc95deb8432cf64353', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 16 00:43:01.049870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81486) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 16 00:43:01.065868 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90eee3a-cd2f-4e69-b4ba-661eb5594eb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:01.066479 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90eee3a-cd2f-4e69-b4ba-661eb5594eb5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:01.066758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90eee3a-cd2f-4e69-b4ba-661eb5594eb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:01.067062 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90eee3a-cd2f-4e69-b4ba-661eb5594eb5 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:01.067736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90eee3a-cd2f-4e69-b4ba-661eb5594eb5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:01.068036 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90eee3a-cd2f-4e69-b4ba-661eb5594eb5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:01.069431 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90eee3a-cd2f-4e69-b4ba-661eb5594eb5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:01.069606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-2b9ecb51-efc9-421f-a334-2cee610320ed None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 16 00:43:01.069606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:01.069606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 16 00:43:01.069606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 16 00:43:01.069606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 16 00:43:01.069606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 16 00:43:01.069606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 16 00:43:01.069606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application result = self._query(query) Oct 16 00:43:01.069606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 16 00:43:01.069606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application conn.query(q) Oct 16 00:43:01.069606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 16 00:43:01.069606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 16 00:43:01.069606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 16 00:43:01.069606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application result.read() Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 16 00:43:01.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 16 00:43:01.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application next(self.gen) Oct 16 00:43:01.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 16 00:43:01.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application yield resource Oct 16 00:43:01.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 16 00:43:01.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application next(self.gen) Oct 16 00:43:01.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 16 00:43:01.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self.session.rollback() Oct 16 00:43:01.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 16 00:43:01.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self.force_reraise() Oct 16 00:43:01.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 16 00:43:01.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise self.value Oct 16 00:43:01.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 16 00:43:01.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 16 00:43:01.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 16 00:43:01.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application session.commit() Oct 16 00:43:01.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 16 00:43:01.071560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 16 00:43:01.072427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 16 00:43:01.072427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._prepare_impl() Oct 16 00:43:01.072427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 16 00:43:01.072427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self.session.flush() Oct 16 00:43:01.072427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 16 00:43:01.072427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._flush(objects) Oct 16 00:43:01.072427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 16 00:43:01.072427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 16 00:43:01.072427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 16 00:43:01.072427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application compat.raise_( Oct 16 00:43:01.072427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 16 00:43:01.072427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception Oct 16 00:43:01.072427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 16 00:43:01.072427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application flush_context.execute() Oct 16 00:43:01.072427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 16 00:43:01.072427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rec.execute(self) Oct 16 00:43:01.072427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 16 00:43:01.073327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 16 00:43:01.073327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 16 00:43:01.073327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 16 00:43:01.073327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 16 00:43:01.073327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 16 00:43:01.073327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 16 00:43:01.073327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 16 00:43:01.073327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 16 00:43:01.073327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 16 00:43:01.073327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 16 00:43:01.073327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 16 00:43:01.073327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 16 00:43:01.073327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 16 00:43:01.073327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 16 00:43:01.073327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 16 00:43:01.074270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 16 00:43:01.074270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception Oct 16 00:43:01.074270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 16 00:43:01.074270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 16 00:43:01.074270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 16 00:43:01.074270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 16 00:43:01.074270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 16 00:43:01.074270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application result = self._query(query) Oct 16 00:43:01.074270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 16 00:43:01.074270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application conn.query(q) Oct 16 00:43:01.074270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 16 00:43:01.074270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 16 00:43:01.074270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 16 00:43:01.074270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application result.read() Oct 16 00:43:01.074270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 16 00:43:01.074270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 16 00:43:01.074270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 16 00:43:01.075141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 16 00:43:01.075141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 16 00:43:01.075141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 16 00:43:01.075141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 16 00:43:01.075141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 16 00:43:01.075141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 16 00:43:01.075141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 16 00:43:01.075141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [parameters: {'id': 'd2290b771f4348cc95deb8432cf64353', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 16 00:43:01.075141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 16 00:43:01.075141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:01.075141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:43:01.075141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:01.075141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:01.075141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 16 00:43:01.075141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 16 00:43:01.076041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 16 00:43:01.076041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 16 00:43:01.076041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 16 00:43:01.076041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:01.076041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:43:01.076041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:01.076041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:01.076041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:01.076041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:01.076041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:01.076041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:01.076041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:01.076041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:01.076041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:01.076041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:01.076041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:01.076041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:01.076991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 16 00:43:01.076991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 16 00:43:01.076991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:01.076991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:01.076991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 16 00:43:01.076991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 16 00:43:01.076991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:01.076991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:01.076991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 16 00:43:01.076991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 16 00:43:01.076991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 16 00:43:01.076991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:01.076991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 8/15] 10.222.0.25 () {60 vars in 1137 bytes} [Sun Oct 16 00:43:00 2022] POST /identity/v3/domains => generated 190 bytes in 190 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:43:01.076991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9ecb51-efc9-421f-a334-2cee610320ed None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:01.085179 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 8/16] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:01.091809 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90eee3a-cd2f-4e69-b4ba-661eb5594eb5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:01.182363 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9ecb51-efc9-421f-a334-2cee610320ed None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:01.196694 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90eee3a-cd2f-4e69-b4ba-661eb5594eb5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:01.218095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1406069f-550f-4914-9f22-d25a9aa17675 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:01.219021 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1406069f-550f-4914-9f22-d25a9aa17675 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:01.219307 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1406069f-550f-4914-9f22-d25a9aa17675 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:01.219579 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1406069f-550f-4914-9f22-d25a9aa17675 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:01.233902 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc84620-2e81-452e-95e4-5fa1d4f7433e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:01.234559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc84620-2e81-452e-95e4-5fa1d4f7433e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:01.234846 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc84620-2e81-452e-95e4-5fa1d4f7433e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:01.235159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc84620-2e81-452e-95e4-5fa1d4f7433e None admin] PATH_INFO: `/v3/users/68c5eeb505a240daa471059d6635deea` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:01.236341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc84620-2e81-452e-95e4-5fa1d4f7433e None admin] RBAC: Authorizing `identity:get_user(user_id=68c5eeb505a240daa471059d6635deea)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:01.237812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc84620-2e81-452e-95e4-5fa1d4f7433e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:01.239459 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 9/17] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:01 2022] GET /identity/v3/users/68c5eeb505a240daa471059d6635deea => generated 247 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:01.240118 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1406069f-550f-4914-9f22-d25a9aa17675 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:01.241613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1406069f-550f-4914-9f22-d25a9aa17675 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:01.245198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc84620-2e81-452e-95e4-5fa1d4f7433e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:01.259300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-1406069f-550f-4914-9f22-d25a9aa17675 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:43:01.259300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:01.259300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:01.259300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:01.259300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:01.259300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:01.259300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:01.259300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:01.259300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:01.259300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:01.259300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:01.259300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:01.259300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:43:01.259300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:43:01.259300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:43:01.259300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:43:01.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:01.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:01.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:01.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:01.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:01.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:01.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:01.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:01.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:01.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:01.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:01.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:01.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:01.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:01.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:01.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:01.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:43:01.261394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:43:01.261394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:43:01.261394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:01.261394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 9/18] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:43:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:43:01.266716 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:01.266922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1406069f-550f-4914-9f22-d25a9aa17675 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:01.291894 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:01.360015 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc84620-2e81-452e-95e4-5fa1d4f7433e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:01.377463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1406069f-550f-4914-9f22-d25a9aa17675 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:01.404643 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458e2494-8a1e-4e03-b920-50b5cd1d003a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:01.405717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458e2494-8a1e-4e03-b920-50b5cd1d003a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:01.406206 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458e2494-8a1e-4e03-b920-50b5cd1d003a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:01.406657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458e2494-8a1e-4e03-b920-50b5cd1d003a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:01.408418 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458e2494-8a1e-4e03-b920-50b5cd1d003a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:01.410853 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458e2494-8a1e-4e03-b920-50b5cd1d003a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:01.419093 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 10/19] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:43:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:01.419475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04daab37-0225-438a-ade9-ccf1b4656658 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:01.420116 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04daab37-0225-438a-ade9-ccf1b4656658 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:01.420402 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04daab37-0225-438a-ade9-ccf1b4656658 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:01.420514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458e2494-8a1e-4e03-b920-50b5cd1d003a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:01.420680 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04daab37-0225-438a-ade9-ccf1b4656658 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:01.421342 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04daab37-0225-438a-ade9-ccf1b4656658 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:01.421646 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04daab37-0225-438a-ade9-ccf1b4656658 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:01.423082 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04daab37-0225-438a-ade9-ccf1b4656658 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:01.443042 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 10/20] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:43:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:01.515659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:01.537994 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458e2494-8a1e-4e03-b920-50b5cd1d003a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:01.576238 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89dfdfd-df69-44b9-b137-82a5f275074c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:01.576940 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89dfdfd-df69-44b9-b137-82a5f275074c None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:01.577226 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89dfdfd-df69-44b9-b137-82a5f275074c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:01.577509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89dfdfd-df69-44b9-b137-82a5f275074c None admin] PATH_INFO: `/v3/domains/default/users/68c5eeb505a240daa471059d6635deea/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:01.579890 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89dfdfd-df69-44b9-b137-82a5f275074c None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=68c5eeb505a240daa471059d6635deea, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:01.581381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89dfdfd-df69-44b9-b137-82a5f275074c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:01.698148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 11/21] 10.222.0.25 () {58 vars in 1361 bytes} [Sun Oct 16 00:43:01 2022] PUT /identity/v3/domains/default/users/68c5eeb505a240daa471059d6635deea/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:43:03.965376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd99108-da79-4cf4-a61b-0d9a21af1285 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:03.965924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd99108-da79-4cf4-a61b-0d9a21af1285 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:03.966001 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd99108-da79-4cf4-a61b-0d9a21af1285 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:03.967748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 11/22] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:03.973942 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fa32ea-eec2-4c62-84d6-d4ec3ce83698 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:03.974427 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fa32ea-eec2-4c62-84d6-d4ec3ce83698 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:03.974501 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fa32ea-eec2-4c62-84d6-d4ec3ce83698 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:04.003292 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-67fa32ea-eec2-4c62-84d6-d4ec3ce83698 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:04.004947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-67fa32ea-eec2-4c62-84d6-d4ec3ce83698 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:04.112320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 12/23] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:04.139617 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b533a9-1fde-42be-87fc-710d050afaec None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:04.139917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b533a9-1fde-42be-87fc-710d050afaec None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:04.140195 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b533a9-1fde-42be-87fc-710d050afaec None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:04.141535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 12/24] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:43:04.147261 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768da4f7-3af1-42e0-8896-6ae0c966fda4 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:04.147559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768da4f7-3af1-42e0-8896-6ae0c966fda4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:04.147839 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768da4f7-3af1-42e0-8896-6ae0c966fda4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:04.176190 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-768da4f7-3af1-42e0-8896-6ae0c966fda4 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:04.177578 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-768da4f7-3af1-42e0-8896-6ae0c966fda4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:04.265694 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:04.270779 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 13/25] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:04.510421 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c2f82d-3d4b-414b-bf68-869a6462e2e1 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:04.510713 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c2f82d-3d4b-414b-bf68-869a6462e2e1 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:04.511212 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c2f82d-3d4b-414b-bf68-869a6462e2e1 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:04.513628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 13/26] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:04.519048 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66135839-8e30-431b-b1b1-fd5577dfb59b None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:04.519609 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66135839-8e30-431b-b1b1-fd5577dfb59b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:04.520146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66135839-8e30-431b-b1b1-fd5577dfb59b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:04.545222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-66135839-8e30-431b-b1b1-fd5577dfb59b None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:04.546792 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-66135839-8e30-431b-b1b1-fd5577dfb59b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:04.653372 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c2f82d-3d4b-414b-bf68-869a6462e2e1 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:04.700181 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 14/27] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:04.784781 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:04.823911 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c2f82d-3d4b-414b-bf68-869a6462e2e1 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:04.836198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66135839-8e30-431b-b1b1-fd5577dfb59b None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:04.953246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5729940-8e2c-4f0b-a466-ee16285747c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:04.955164 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5729940-8e2c-4f0b-a466-ee16285747c3 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:04.956841 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5729940-8e2c-4f0b-a466-ee16285747c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:04.957711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5729940-8e2c-4f0b-a466-ee16285747c3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:04.995137 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5729940-8e2c-4f0b-a466-ee16285747c3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:05.000182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5729940-8e2c-4f0b-a466-ee16285747c3 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:05.040567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-a5729940-8e2c-4f0b-a466-ee16285747c3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:43:05.040567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:05.040567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:05.040567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:05.040567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:05.040567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:05.040567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:05.040567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:05.040567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:05.040567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:05.040567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:05.040567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:05.040567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:43:05.040567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:43:05.040567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:43:05.040567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:43:05.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:05.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:05.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:05.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:05.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:05.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:05.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:05.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:05.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:05.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:05.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:05.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:05.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:05.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:05.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:05.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:05.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:43:05.045349 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:43:05.045349 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:43:05.045349 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:05.045349 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 14/28] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:43:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 389 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:43:05.048926 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5729940-8e2c-4f0b-a466-ee16285747c3 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:05.205246 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66135839-8e30-431b-b1b1-fd5577dfb59b None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:05.210335 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5729940-8e2c-4f0b-a466-ee16285747c3 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:05.245368 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:05.251122 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9fbbe4-8d23-4eae-ad40-215ed3a0f358 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:05.251766 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9fbbe4-8d23-4eae-ad40-215ed3a0f358 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:05.252191 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9fbbe4-8d23-4eae-ad40-215ed3a0f358 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:05.252346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9fbbe4-8d23-4eae-ad40-215ed3a0f358 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:05.252994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9fbbe4-8d23-4eae-ad40-215ed3a0f358 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:05.253407 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9fbbe4-8d23-4eae-ad40-215ed3a0f358 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:05.254739 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9fbbe4-8d23-4eae-ad40-215ed3a0f358 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:05.269591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 15/29] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:43:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:05.271022 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9fbbe4-8d23-4eae-ad40-215ed3a0f358 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:05.309617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:05.323024 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8992de-c7b4-43d0-8a0d-8216f64902c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:05.324608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8992de-c7b4-43d0-8a0d-8216f64902c9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:05.325103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8992de-c7b4-43d0-8a0d-8216f64902c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:05.325598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8992de-c7b4-43d0-8a0d-8216f64902c9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:05.376465 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8992de-c7b4-43d0-8a0d-8216f64902c9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:05.384705 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9fbbe4-8d23-4eae-ad40-215ed3a0f358 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:05.395112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-3d8992de-c7b4-43d0-8a0d-8216f64902c9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:43:05.395112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:05.395112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:05.395112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:05.395112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:05.395112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:05.395112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:05.395112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:05.395112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:05.395112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:05.395112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:05.395112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:05.395112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:05.395112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:05.395112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:05.395112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:05.396020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:05.396020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:05.396020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:05.396020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:05.396020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:05.396020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:05.396020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:05.396020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:05.396020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:05.396020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:05.396020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:05.396020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:05.396020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:05.396020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:05.396020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:05.396020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:05.396818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:05.396818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:05.396818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:05.396818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:05.396818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:05.396818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:05.396818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:05.396818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:05.396818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:05.396818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:05.396818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:43:05.396818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:05.397392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 15/30] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:43:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 562 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:05.398433 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8992de-c7b4-43d0-8a0d-8216f64902c9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:05.427648 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f488e1-fa92-44c2-8a59-c5832a4674a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:05.428346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f488e1-fa92-44c2-8a59-c5832a4674a1 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:05.428704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f488e1-fa92-44c2-8a59-c5832a4674a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:05.428988 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f488e1-fa92-44c2-8a59-c5832a4674a1 None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:05.430095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f488e1-fa92-44c2-8a59-c5832a4674a1 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:05.431757 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f488e1-fa92-44c2-8a59-c5832a4674a1 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:05.596006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 16/31] 10.222.0.25 () {60 vars in 1131 bytes} [Sun Oct 16 00:43:05 2022] POST /identity/v3/roles => generated 219 bytes in 326 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:43:05.604196 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f488e1-fa92-44c2-8a59-c5832a4674a1 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:05.693125 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8992de-c7b4-43d0-8a0d-8216f64902c9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:05.769652 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:05.826022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d16ce7-e35f-4858-a837-46a508448f01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:05.827041 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d16ce7-e35f-4858-a837-46a508448f01 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:05.827497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d16ce7-e35f-4858-a837-46a508448f01 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:05.827976 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d16ce7-e35f-4858-a837-46a508448f01 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:05.828812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d16ce7-e35f-4858-a837-46a508448f01 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:05.830956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d16ce7-e35f-4858-a837-46a508448f01 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:05.932102 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f488e1-fa92-44c2-8a59-c5832a4674a1 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:05.933273 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 16/32] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 16 00:43:05 2022] POST /identity/v3/projects => generated 301 bytes in 535 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:43:05.996095 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27efcaf7-70f4-4958-999c-fde04bf13a1f None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:05.997045 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27efcaf7-70f4-4958-999c-fde04bf13a1f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:05.997802 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27efcaf7-70f4-4958-999c-fde04bf13a1f None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:06.005418 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80cfb03a-3d85-4aef-95cc-bbe16e01de77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:06.006248 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cfb03a-3d85-4aef-95cc-bbe16e01de77 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:06.006422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cfb03a-3d85-4aef-95cc-bbe16e01de77 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:06.006798 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cfb03a-3d85-4aef-95cc-bbe16e01de77 None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:06.007372 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80cfb03a-3d85-4aef-95cc-bbe16e01de77 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:06.007693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80cfb03a-3d85-4aef-95cc-bbe16e01de77 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:06.007841 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 17/33] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:05 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:06.009304 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80cfb03a-3d85-4aef-95cc-bbe16e01de77 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:06.020171 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b9b2bf-7a67-4bc2-ac52-f031493618ce None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:06.020418 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b9b2bf-7a67-4bc2-ac52-f031493618ce None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:06.020743 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b9b2bf-7a67-4bc2-ac52-f031493618ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:06.038699 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 17/34] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 442 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:06.048120 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80cfb03a-3d85-4aef-95cc-bbe16e01de77 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:06.062437 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d6b9b2bf-7a67-4bc2-ac52-f031493618ce None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:06.063755 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d6b9b2bf-7a67-4bc2-ac52-f031493618ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:06.171604 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:06.241379 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 18/35] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:06.263943 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80cfb03a-3d85-4aef-95cc-bbe16e01de77 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:06.282089 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45069ca6-9956-4eac-935a-7d06db697f45 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:06.282393 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45069ca6-9956-4eac-935a-7d06db697f45 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:06.282680 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45069ca6-9956-4eac-935a-7d06db697f45 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:06.284335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 19/36] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:43:06.291248 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:06.296365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca14266e-e1a9-448f-b315-4fd67f37e640 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:06.296733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca14266e-e1a9-448f-b315-4fd67f37e640 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:06.297014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca14266e-e1a9-448f-b315-4fd67f37e640 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:06.344091 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ca14266e-e1a9-448f-b315-4fd67f37e640 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:06.345617 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b70bc5-e158-48ab-8603-d6b93a9e6ff5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:06.345831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ca14266e-e1a9-448f-b315-4fd67f37e640 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:06.347272 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b70bc5-e158-48ab-8603-d6b93a9e6ff5 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:06.348060 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b70bc5-e158-48ab-8603-d6b93a9e6ff5 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:06.348832 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b70bc5-e158-48ab-8603-d6b93a9e6ff5 None admin] PATH_INFO: `/v3/users/68c5eeb505a240daa471059d6635deea` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:06.353484 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b70bc5-e158-48ab-8603-d6b93a9e6ff5 None admin] RBAC: Authorizing `identity:get_user(user_id=68c5eeb505a240daa471059d6635deea)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:06.357452 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b70bc5-e158-48ab-8603-d6b93a9e6ff5 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:06.360943 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 18/37] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:06 2022] GET /identity/v3/users/68c5eeb505a240daa471059d6635deea => generated 247 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:06.369824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b70bc5-e158-48ab-8603-d6b93a9e6ff5 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:06.541174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 20/38] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:06.577772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca14266e-e1a9-448f-b315-4fd67f37e640 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:06.599465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b70bc5-e158-48ab-8603-d6b93a9e6ff5 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:06.659647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458da070-2fb8-492e-b4a2-67d869556815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:06.661040 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458da070-2fb8-492e-b4a2-67d869556815 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:06.661733 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458da070-2fb8-492e-b4a2-67d869556815 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:06.662411 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458da070-2fb8-492e-b4a2-67d869556815 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:06.664765 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458da070-2fb8-492e-b4a2-67d869556815 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:06.668672 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458da070-2fb8-492e-b4a2-67d869556815 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:06.672209 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 19/39] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:43:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:06.680555 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458da070-2fb8-492e-b4a2-67d869556815 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:06.745519 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:06.781821 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca14266e-e1a9-448f-b315-4fd67f37e640 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:06.795451 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:06.824434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9d0525-ecff-4724-8bfb-3e8ae5e99b2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:06.825619 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9d0525-ecff-4724-8bfb-3e8ae5e99b2e None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:06.826148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9d0525-ecff-4724-8bfb-3e8ae5e99b2e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:06.826643 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9d0525-ecff-4724-8bfb-3e8ae5e99b2e None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:06.827577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9d0525-ecff-4724-8bfb-3e8ae5e99b2e None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:06.829199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9d0525-ecff-4724-8bfb-3e8ae5e99b2e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:06.848888 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 21/40] 10.222.0.25 () {60 vars in 1131 bytes} [Sun Oct 16 00:43:06 2022] POST /identity/v3/roles => generated 225 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:43:06.876099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458da070-2fb8-492e-b4a2-67d869556815 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:06.920817 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c585d3b-a41a-449e-8708-b5feb7fe3f6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:06.922023 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c585d3b-a41a-449e-8708-b5feb7fe3f6e None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:06.922517 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c585d3b-a41a-449e-8708-b5feb7fe3f6e None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:06.923056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c585d3b-a41a-449e-8708-b5feb7fe3f6e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:06.924025 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c585d3b-a41a-449e-8708-b5feb7fe3f6e None admin] RBAC: Adding query filter params (role.id=c5cd777d70a649028f970fd986dc484f, scope.domain.id=default, user.id=68c5eeb505a240daa471059d6635deea) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:06.924687 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c585d3b-a41a-449e-8708-b5feb7fe3f6e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:06.926408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c585d3b-a41a-449e-8708-b5feb7fe3f6e None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:06.938975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 20/41] 10.222.0.25 () {56 vars in 1322 bytes} [Sun Oct 16 00:43:06 2022] GET /identity/v3/role_assignments?user.id=68c5eeb505a240daa471059d6635deea&scope.domain.id=default&role.id=c5cd777d70a649028f970fd986dc484f => generated 530 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:07.059280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9d0525-ecff-4724-8bfb-3e8ae5e99b2e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:07.086255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012506cc-0d01-42ec-a14f-aac4ddf072ae None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:07.087356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012506cc-0d01-42ec-a14f-aac4ddf072ae None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:07.087862 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012506cc-0d01-42ec-a14f-aac4ddf072ae None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:07.090073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 21/42] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:07 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:07.097296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146b240e-a110-493d-8159-8cd2dd669579 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:07.097967 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146b240e-a110-493d-8159-8cd2dd669579 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:07.098583 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146b240e-a110-493d-8159-8cd2dd669579 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:07.123314 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-146b240e-a110-493d-8159-8cd2dd669579 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:07.125463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-146b240e-a110-493d-8159-8cd2dd669579 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:07.241638 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 22/43] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:43:07.321738 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9d0525-ecff-4724-8bfb-3e8ae5e99b2e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:07.363923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be566e35-a6d3-4f96-9d4e-b6a56a571025 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:07.364580 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be566e35-a6d3-4f96-9d4e-b6a56a571025 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:07.364866 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be566e35-a6d3-4f96-9d4e-b6a56a571025 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:07.365146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be566e35-a6d3-4f96-9d4e-b6a56a571025 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:07.366304 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be566e35-a6d3-4f96-9d4e-b6a56a571025 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:07.367815 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be566e35-a6d3-4f96-9d4e-b6a56a571025 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:07.369456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 22/44] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:43:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:07.375684 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146b240e-a110-493d-8159-8cd2dd669579 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:07.430800 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:07.489587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146b240e-a110-493d-8159-8cd2dd669579 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:07.527714 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a8e502-30a9-48e4-9bac-f781948e6696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:07.528358 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a8e502-30a9-48e4-9bac-f781948e6696 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:07.528647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a8e502-30a9-48e4-9bac-f781948e6696 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:07.528929 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a8e502-30a9-48e4-9bac-f781948e6696 None admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:07.529598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a8e502-30a9-48e4-9bac-f781948e6696 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:07.531098 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a8e502-30a9-48e4-9bac-f781948e6696 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:07.561898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 23/45] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 16 00:43:07 2022] POST /identity/v3/projects => generated 312 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:43:07.755156 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be566e35-a6d3-4f96-9d4e-b6a56a571025 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:07.807174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:07.889791 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be566e35-a6d3-4f96-9d4e-b6a56a571025 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:07.926123 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fa7bce-71f7-46a8-bcfe-bb8ad5daaeed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:07.926769 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fa7bce-71f7-46a8-bcfe-bb8ad5daaeed None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:07.927086 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fa7bce-71f7-46a8-bcfe-bb8ad5daaeed None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:07.927369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fa7bce-71f7-46a8-bcfe-bb8ad5daaeed None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:07.928079 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fa7bce-71f7-46a8-bcfe-bb8ad5daaeed None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:07.929810 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fa7bce-71f7-46a8-bcfe-bb8ad5daaeed None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:07.948145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 23/46] 10.222.0.25 () {60 vars in 1131 bytes} [Sun Oct 16 00:43:07 2022] POST /identity/v3/roles => generated 223 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:43:09.910691 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8539b5-7a9b-4d82-9b64-1aba5fa12aeb None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:09.911246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8539b5-7a9b-4d82-9b64-1aba5fa12aeb None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:09.911355 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8539b5-7a9b-4d82-9b64-1aba5fa12aeb None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:09.912653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 24/47] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:09.918599 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f63e4a-91de-463e-8479-2c392edd9e34 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:09.919067 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f63e4a-91de-463e-8479-2c392edd9e34 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:09.919183 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f63e4a-91de-463e-8479-2c392edd9e34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:09.937762 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a7f63e4a-91de-463e-8479-2c392edd9e34 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:09.939025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a7f63e4a-91de-463e-8479-2c392edd9e34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:10.008350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 24/48] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:10.272348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8539b5-7a9b-4d82-9b64-1aba5fa12aeb None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:10.367403 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:10.375895 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8539b5-7a9b-4d82-9b64-1aba5fa12aeb None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:10.409612 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f7a5db-9bbe-4205-b757-b8100ba3cb71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:10.410255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f7a5db-9bbe-4205-b757-b8100ba3cb71 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:10.410542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f7a5db-9bbe-4205-b757-b8100ba3cb71 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:10.410824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f7a5db-9bbe-4205-b757-b8100ba3cb71 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:10.411974 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f7a5db-9bbe-4205-b757-b8100ba3cb71 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:10.413446 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f7a5db-9bbe-4205-b757-b8100ba3cb71 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:10.415048 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 25/49] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:43:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:10.421136 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f63e4a-91de-463e-8479-2c392edd9e34 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:10.442263 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:10.520756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f63e4a-91de-463e-8479-2c392edd9e34 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:10.554012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a438731-91b0-4dc8-b493-a74d82340c66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:10.554626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a438731-91b0-4dc8-b493-a74d82340c66 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:10.554940 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a438731-91b0-4dc8-b493-a74d82340c66 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:10.555219 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a438731-91b0-4dc8-b493-a74d82340c66 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:10.555876 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a438731-91b0-4dc8-b493-a74d82340c66 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:10.557332 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a438731-91b0-4dc8-b493-a74d82340c66 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:10.583164 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 25/50] 10.222.0.25 () {60 vars in 1140 bytes} [Sun Oct 16 00:43:10 2022] POST /identity/v3/projects => generated 298 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:43:12.783380 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb6b520-c49d-4c99-8c95-ca686482102e None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:12.783893 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb6b520-c49d-4c99-8c95-ca686482102e None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:12.783893 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb6b520-c49d-4c99-8c95-ca686482102e None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:12.785204 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 26/51] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:12.790562 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b13f979-4fc0-47bf-9de2-b7e42c12ae9c None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:12.790998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b13f979-4fc0-47bf-9de2-b7e42c12ae9c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:12.791168 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b13f979-4fc0-47bf-9de2-b7e42c12ae9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:12.811085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3b13f979-4fc0-47bf-9de2-b7e42c12ae9c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:12.812297 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3b13f979-4fc0-47bf-9de2-b7e42c12ae9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:12.881255 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 26/52] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:43:13.160146 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb6b520-c49d-4c99-8c95-ca686482102e None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:13.262790 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb6b520-c49d-4c99-8c95-ca686482102e None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:13.295243 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04299f63-6f5a-4ac6-b835-4749fcd026a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:13.295880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04299f63-6f5a-4ac6-b835-4749fcd026a4 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:13.296167 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04299f63-6f5a-4ac6-b835-4749fcd026a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:13.296447 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04299f63-6f5a-4ac6-b835-4749fcd026a4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:13.297569 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04299f63-6f5a-4ac6-b835-4749fcd026a4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:13.299063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04299f63-6f5a-4ac6-b835-4749fcd026a4 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:13.300640 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 27/53] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:43:13 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:13.306204 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b13f979-4fc0-47bf-9de2-b7e42c12ae9c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:13.309035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:13.406617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b13f979-4fc0-47bf-9de2-b7e42c12ae9c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:13.441703 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac95bb7-8698-46d7-856b-a610b8ded642 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:13.442324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac95bb7-8698-46d7-856b-a610b8ded642 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:13.442612 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac95bb7-8698-46d7-856b-a610b8ded642 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:13.442913 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac95bb7-8698-46d7-856b-a610b8ded642 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:13.443565 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac95bb7-8698-46d7-856b-a610b8ded642 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:13.445029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac95bb7-8698-46d7-856b-a610b8ded642 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:13.471253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 27/54] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:43:13 2022] POST /identity/v3/users => generated 275 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:43:16.778678 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955fe5ac-f18d-4aa2-ab12-17a5ae98492d None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:16.779700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955fe5ac-f18d-4aa2-ab12-17a5ae98492d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:16.780217 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955fe5ac-f18d-4aa2-ab12-17a5ae98492d None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:16.781805 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 28/55] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:43:16.787943 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb9037a-f47d-47e1-98c4-4650ab8dd9b7 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:16.788879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb9037a-f47d-47e1-98c4-4650ab8dd9b7 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:16.789355 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb9037a-f47d-47e1-98c4-4650ab8dd9b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:16.816132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2cb9037a-f47d-47e1-98c4-4650ab8dd9b7 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:16.817736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2cb9037a-f47d-47e1-98c4-4650ab8dd9b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:16.899577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:16.929338 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 28/56] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:16.986883 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1e0973-b036-4190-a789-094165e728b1 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:16.987228 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1e0973-b036-4190-a789-094165e728b1 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:16.987453 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1e0973-b036-4190-a789-094165e728b1 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:16.988825 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 29/57] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:16.994631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a4d6a7-c820-46bc-b99d-18c9d00cd198 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:16.994952 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a4d6a7-c820-46bc-b99d-18c9d00cd198 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:16.995239 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a4d6a7-c820-46bc-b99d-18c9d00cd198 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:17.021359 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-57a4d6a7-c820-46bc-b99d-18c9d00cd198 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:17.022594 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-57a4d6a7-c820-46bc-b99d-18c9d00cd198 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:17.141069 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 29/58] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:17.233640 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1e0973-b036-4190-a789-094165e728b1 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:17.282081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:17.367234 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c13e5a-b484-4343-bc1c-a2f2a1b4b3c5 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:17.367857 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c13e5a-b484-4343-bc1c-a2f2a1b4b3c5 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:17.368362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c13e5a-b484-4343-bc1c-a2f2a1b4b3c5 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:17.369961 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 30/59] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:17.375859 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e89d7e-4739-4fdb-a564-5bda8aca32cd None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:17.376450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e89d7e-4739-4fdb-a564-5bda8aca32cd None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:17.376948 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e89d7e-4739-4fdb-a564-5bda8aca32cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:17.390812 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1e0973-b036-4190-a789-094165e728b1 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:17.397812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e4e89d7e-4739-4fdb-a564-5bda8aca32cd None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:17.402697 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e4e89d7e-4739-4fdb-a564-5bda8aca32cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:17.460020 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e744ed-d020-4f6b-8549-f6023f099bb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:17.461533 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e744ed-d020-4f6b-8549-f6023f099bb8 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:17.462336 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e744ed-d020-4f6b-8549-f6023f099bb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:17.462943 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e744ed-d020-4f6b-8549-f6023f099bb8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:17.478141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e744ed-d020-4f6b-8549-f6023f099bb8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:17.487412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-a4e744ed-d020-4f6b-8549-f6023f099bb8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:43:17.487412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:17.487412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:17.487412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:17.487412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:17.487412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:17.487412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:17.487412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:17.487412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:17.487412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:17.487412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:17.487412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:17.487412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:17.487412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:17.487412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:17.487412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:17.488378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:17.488378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:17.488378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:17.488378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:17.488378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:17.488378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:17.488378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:17.488378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:17.488378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:17.488378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:17.488378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:17.488378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:17.488378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:17.488378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:17.488378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:17.488378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:17.489176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:17.489176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:17.489176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:17.489176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:17.489176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:17.489176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:17.489176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:17.489176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:17.489176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:17.489176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:17.489176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:43:17.489176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:17.493075 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 30/60] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:43:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:17.494521 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8fd137-06eb-4c41-848b-7716cac08755 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:17.494818 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8fd137-06eb-4c41-848b-7716cac08755 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:17.495128 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8fd137-06eb-4c41-848b-7716cac08755 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:17.501259 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 31/61] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:17.501259 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8fd137-06eb-4c41-848b-7716cac08755 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:17.519234 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 31/62] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:17.521245 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e50c3c0-eb21-4974-af31-21f6725e4db7 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:17.521827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e50c3c0-eb21-4974-af31-21f6725e4db7 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:17.522216 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e50c3c0-eb21-4974-af31-21f6725e4db7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:17.543424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5e50c3c0-eb21-4974-af31-21f6725e4db7 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:17.544792 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5e50c3c0-eb21-4974-af31-21f6725e4db7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:17.598770 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:17.631618 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:17.635081 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 32/63] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:17.636371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e50c3c0-eb21-4974-af31-21f6725e4db7 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:17.646984 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8fd137-06eb-4c41-848b-7716cac08755 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:17.686518 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499804df-6c42-482a-af29-5909f531c222 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:17.687476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499804df-6c42-482a-af29-5909f531c222 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:17.687948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499804df-6c42-482a-af29-5909f531c222 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:17.688434 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499804df-6c42-482a-af29-5909f531c222 None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:17.689305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499804df-6c42-482a-af29-5909f531c222 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:17.689842 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499804df-6c42-482a-af29-5909f531c222 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:17.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499804df-6c42-482a-af29-5909f531c222 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:17.700865 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 32/64] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:17.702541 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e49c19-9e12-46e6-8daf-e29439f8bc54 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:17.703043 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e49c19-9e12-46e6-8daf-e29439f8bc54 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:17.703502 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e49c19-9e12-46e6-8daf-e29439f8bc54 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:17.705246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 33/65] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:17.710855 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e49c19-9e12-46e6-8daf-e29439f8bc54 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:17.759726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e50c3c0-eb21-4974-af31-21f6725e4db7 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:17.802657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4e7be3-a0e2-4214-a5a1-adac717e7b00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:17.803383 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4e7be3-a0e2-4214-a5a1-adac717e7b00 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:17.803663 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4e7be3-a0e2-4214-a5a1-adac717e7b00 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:17.803942 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4e7be3-a0e2-4214-a5a1-adac717e7b00 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:17.817453 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4e7be3-a0e2-4214-a5a1-adac717e7b00 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:17.822595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-bd4e7be3-a0e2-4214-a5a1-adac717e7b00 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:17.822595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:17.822595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:17.822595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:17.822595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:17.822595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:17.822595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:17.822595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:17.822595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:17.822595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:17.822595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:17.822595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:17.822595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:17.822595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:17.822595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:17.822595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:17.823696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:17.823696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:17.823696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:17.823696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:17.823696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:17.823696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:17.823696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:17.823696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:17.823696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:17.823696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:17.823696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:17.823696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:17.823696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:17.823696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:17.823696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:17.823696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:17.824708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:17.824708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:17.824708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:17.824708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:17.824708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:17.824708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:17.824708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:17.824708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:17.824708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:17.824708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:17.824708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:17.824708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:17.824708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 33/66] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:17.831167 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20310cf-fa1e-4188-9ebd-df02cbd3fa87 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:17.831167 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20310cf-fa1e-4188-9ebd-df02cbd3fa87 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:17.831167 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20310cf-fa1e-4188-9ebd-df02cbd3fa87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:17.847093 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d20310cf-fa1e-4188-9ebd-df02cbd3fa87 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:17.848225 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d20310cf-fa1e-4188-9ebd-df02cbd3fa87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:17.882353 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:17.925155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 34/67] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:17.926056 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20310cf-fa1e-4188-9ebd-df02cbd3fa87 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:17.942503 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e49c19-9e12-46e6-8daf-e29439f8bc54 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:17.984465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517f7b06-0def-437e-be79-d00180410561 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:17.985221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517f7b06-0def-437e-be79-d00180410561 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:17.985511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517f7b06-0def-437e-be79-d00180410561 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:17.985788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517f7b06-0def-437e-be79-d00180410561 None admin] PATH_INFO: `/v3/users/68c5eeb505a240daa471059d6635deea` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:17.986973 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517f7b06-0def-437e-be79-d00180410561 None admin] RBAC: Authorizing `identity:get_user(user_id=68c5eeb505a240daa471059d6635deea)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:17.988458 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517f7b06-0def-437e-be79-d00180410561 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:17.990227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 34/68] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:17 2022] GET /identity/v3/users/68c5eeb505a240daa471059d6635deea => generated 247 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:17.990963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517f7b06-0def-437e-be79-d00180410561 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:18.017782 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:18.044590 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20310cf-fa1e-4188-9ebd-df02cbd3fa87 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:18.082426 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d33f84d-26e6-4347-89c7-6d34872b1a23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:18.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d33f84d-26e6-4347-89c7-6d34872b1a23 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:18.083384 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d33f84d-26e6-4347-89c7-6d34872b1a23 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:18.083664 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d33f84d-26e6-4347-89c7-6d34872b1a23 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:18.084352 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d33f84d-26e6-4347-89c7-6d34872b1a23 None admin] RBAC: Adding query filter params (name=member) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:18.084659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d33f84d-26e6-4347-89c7-6d34872b1a23 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:18.086073 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d33f84d-26e6-4347-89c7-6d34872b1a23 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:18.101044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 35/69] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:43:17 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:18.101044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d33f84d-26e6-4347-89c7-6d34872b1a23 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:18.104394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517f7b06-0def-437e-be79-d00180410561 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:18.134228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:18.143563 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f800bf8-14a8-4345-8e6a-cf601cd480ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:18.144246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f800bf8-14a8-4345-8e6a-cf601cd480ef None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:18.144533 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f800bf8-14a8-4345-8e6a-cf601cd480ef None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:18.144812 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f800bf8-14a8-4345-8e6a-cf601cd480ef None admin] PATH_INFO: `/v3/roles/member` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:18.156174 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f800bf8-14a8-4345-8e6a-cf601cd480ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:18.161495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-4f800bf8-14a8-4345-8e6a-cf601cd480ef None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:18.161495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:18.161495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:18.161495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:18.161495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:18.161495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:18.161495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:18.161495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:18.161495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:18.161495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:18.161495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:18.161495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:18.161495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:18.161495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:18.161495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:18.161495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:18.162391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:18.162391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:18.162391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:18.162391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:18.162391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:18.162391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:18.162391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:18.162391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:18.162391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:18.162391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:18.162391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:18.162391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:18.162391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:18.162391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:18.162391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:18.162391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:18.168686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:18.168686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:18.168686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:18.168686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:18.168686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:18.168686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:18.168686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:18.168686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:18.168686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:18.168686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:18.168686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:18.168686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:18.168686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 35/70] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:18.168686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f800bf8-14a8-4345-8e6a-cf601cd480ef None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:18.216779 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d33f84d-26e6-4347-89c7-6d34872b1a23 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:18.251156 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd1d8a6-0ffc-4b76-ad1f-5d9f3f8acb5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:18.251799 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd1d8a6-0ffc-4b76-ad1f-5d9f3f8acb5a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:18.252093 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd1d8a6-0ffc-4b76-ad1f-5d9f3f8acb5a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:18.252372 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd1d8a6-0ffc-4b76-ad1f-5d9f3f8acb5a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:18.255572 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:18.264045 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd1d8a6-0ffc-4b76-ad1f-5d9f3f8acb5a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:18.269029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-8cd1d8a6-0ffc-4b76-ad1f-5d9f3f8acb5a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:43:18.269029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:18.269029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:18.269029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:18.269029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:18.269029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:18.269029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:18.269029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:18.269029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:18.269029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:18.269029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:18.269029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:18.269029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:18.269029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:18.269029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:18.269029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:18.270074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:18.270074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:18.270074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:18.270074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:18.270074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:18.270074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:18.270074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:18.270074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:18.270074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:18.270074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:18.270074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:18.270074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:18.270074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:18.270074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:18.270074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:18.270074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:18.270882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:18.270882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:18.270882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:18.270882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:18.270882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:18.270882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:18.270882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:18.270882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:18.270882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:18.270882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:18.270882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:43:18.270882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:18.270882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 36/71] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 16 00:43:18 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:18.271544 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd1d8a6-0ffc-4b76-ad1f-5d9f3f8acb5a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:18.278967 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f800bf8-14a8-4345-8e6a-cf601cd480ef None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:18.313357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4976ce8e-acb0-4466-8953-4b4bc812d874 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:18.314006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4976ce8e-acb0-4466-8953-4b4bc812d874 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:18.314293 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4976ce8e-acb0-4466-8953-4b4bc812d874 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:18.314572 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4976ce8e-acb0-4466-8953-4b4bc812d874 None admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:18.315764 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4976ce8e-acb0-4466-8953-4b4bc812d874 None admin] RBAC: Authorizing `identity:get_project(project_id=c3343320b0624293946f3195f2d767e2)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:18.317230 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4976ce8e-acb0-4466-8953-4b4bc812d874 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:18.318885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 36/72] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:18 2022] GET /identity/v3/projects/c3343320b0624293946f3195f2d767e2 => generated 298 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:18.319768 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4976ce8e-acb0-4466-8953-4b4bc812d874 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:18.365354 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:18.377079 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd1d8a6-0ffc-4b76-ad1f-5d9f3f8acb5a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:18.415788 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c46e8b0-b3c7-4645-9a67-83ba9d45b574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:18.416453 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c46e8b0-b3c7-4645-9a67-83ba9d45b574 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:18.416752 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c46e8b0-b3c7-4645-9a67-83ba9d45b574 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:18.417038 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c46e8b0-b3c7-4645-9a67-83ba9d45b574 None admin] PATH_INFO: `/v3/users/620e6f7dab50405d9bebd872ceb59dc7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:18.432090 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4976ce8e-acb0-4466-8953-4b4bc812d874 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:18.432300 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c46e8b0-b3c7-4645-9a67-83ba9d45b574 None admin] RBAC: Authorizing `identity:get_user(user_id=620e6f7dab50405d9bebd872ceb59dc7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:18.434281 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c46e8b0-b3c7-4645-9a67-83ba9d45b574 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:18.436111 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 37/73] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:18 2022] GET /identity/v3/users/620e6f7dab50405d9bebd872ceb59dc7 => generated 275 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:18.436961 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c46e8b0-b3c7-4645-9a67-83ba9d45b574 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:18.469949 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f68a60-01c6-46bc-b2b5-568694d32f1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:18.470466 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f68a60-01c6-46bc-b2b5-568694d32f1b None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:18.470757 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f68a60-01c6-46bc-b2b5-568694d32f1b None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:18.471074 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f68a60-01c6-46bc-b2b5-568694d32f1b None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:18.471751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f68a60-01c6-46bc-b2b5-568694d32f1b None admin] RBAC: Adding query filter params (name=member) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:18.472060 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f68a60-01c6-46bc-b2b5-568694d32f1b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:18.473463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f68a60-01c6-46bc-b2b5-568694d32f1b None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:18.482440 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 37/74] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:43:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:18.483322 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f68a60-01c6-46bc-b2b5-568694d32f1b None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:18.504272 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:18.539811 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c46e8b0-b3c7-4645-9a67-83ba9d45b574 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:18.575121 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2061778c-25cd-4443-b2f4-c7afc0829b4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:18.575761 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2061778c-25cd-4443-b2f4-c7afc0829b4b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:18.576046 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2061778c-25cd-4443-b2f4-c7afc0829b4b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:18.576313 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2061778c-25cd-4443-b2f4-c7afc0829b4b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:18.577435 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2061778c-25cd-4443-b2f4-c7afc0829b4b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:18.578944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2061778c-25cd-4443-b2f4-c7afc0829b4b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:18.581006 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 38/75] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:43:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:18.581562 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2061778c-25cd-4443-b2f4-c7afc0829b4b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:18.596094 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f68a60-01c6-46bc-b2b5-568694d32f1b None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:18.602340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:18.635245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f4906f-ec09-49df-a584-d5d07bc2d17f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:18.636631 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f4906f-ec09-49df-a584-d5d07bc2d17f None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:18.636928 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f4906f-ec09-49df-a584-d5d07bc2d17f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:18.637205 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f4906f-ec09-49df-a584-d5d07bc2d17f None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:18.637996 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f4906f-ec09-49df-a584-d5d07bc2d17f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:18.638303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f4906f-ec09-49df-a584-d5d07bc2d17f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:18.639714 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f4906f-ec09-49df-a584-d5d07bc2d17f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:18.648149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 38/76] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:43:18 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:18.648819 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f4906f-ec09-49df-a584-d5d07bc2d17f None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:18.696275 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2061778c-25cd-4443-b2f4-c7afc0829b4b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:18.726540 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:18.733186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2575ffb4-5ff1-46a2-a768-564a256344dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:18.733871 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2575ffb4-5ff1-46a2-a768-564a256344dc None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:18.734157 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2575ffb4-5ff1-46a2-a768-564a256344dc None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:18.734445 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2575ffb4-5ff1-46a2-a768-564a256344dc None admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2/users/68c5eeb505a240daa471059d6635deea/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:18.736458 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2575ffb4-5ff1-46a2-a768-564a256344dc None admin] RBAC: Authorizing `identity:create_grant(project_id=c3343320b0624293946f3195f2d767e2, user_id=68c5eeb505a240daa471059d6635deea, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:18.737954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2575ffb4-5ff1-46a2-a768-564a256344dc None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:18.752421 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 39/77] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:43:18 2022] PUT /identity/v3/projects/c3343320b0624293946f3195f2d767e2/users/68c5eeb505a240daa471059d6635deea/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:43:18.753353 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2575ffb4-5ff1-46a2-a768-564a256344dc None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:18.761759 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f4906f-ec09-49df-a584-d5d07bc2d17f None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:18.804401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a559e391-fefa-4e6e-a3c6-580b601ba45b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:18.805073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a559e391-fefa-4e6e-a3c6-580b601ba45b None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:18.805369 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a559e391-fefa-4e6e-a3c6-580b601ba45b None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:18.805654 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a559e391-fefa-4e6e-a3c6-580b601ba45b None admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:18.807023 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a559e391-fefa-4e6e-a3c6-580b601ba45b None admin] RBAC: Authorizing `identity:get_project(project_id=c3343320b0624293946f3195f2d767e2)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:18.810480 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a559e391-fefa-4e6e-a3c6-580b601ba45b None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:18.817421 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 39/78] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:18 2022] GET /identity/v3/projects/c3343320b0624293946f3195f2d767e2 => generated 298 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:18.817421 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a559e391-fefa-4e6e-a3c6-580b601ba45b None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:18.846831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:18.874666 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2575ffb4-5ff1-46a2-a768-564a256344dc None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:18.915350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffd2fe8-df16-4ac9-b789-5594c1f9740a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:18.916079 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffd2fe8-df16-4ac9-b789-5594c1f9740a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:18.916389 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffd2fe8-df16-4ac9-b789-5594c1f9740a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:18.916659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffd2fe8-df16-4ac9-b789-5594c1f9740a None admin] PATH_INFO: `/v3/users/620e6f7dab50405d9bebd872ceb59dc7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:18.917938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffd2fe8-df16-4ac9-b789-5594c1f9740a None admin] RBAC: Authorizing `identity:get_user(user_id=620e6f7dab50405d9bebd872ceb59dc7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:18.919496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffd2fe8-df16-4ac9-b789-5594c1f9740a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:18.926266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 40/79] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:18 2022] GET /identity/v3/users/620e6f7dab50405d9bebd872ceb59dc7 => generated 275 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:18.926266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffd2fe8-df16-4ac9-b789-5594c1f9740a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:18.956491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a559e391-fefa-4e6e-a3c6-580b601ba45b None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:18.994705 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db626824-b4c5-42b9-91a6-d896e4523a0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:18.995305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db626824-b4c5-42b9-91a6-d896e4523a0c None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:18.995766 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db626824-b4c5-42b9-91a6-d896e4523a0c None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:18.996247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db626824-b4c5-42b9-91a6-d896e4523a0c None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:18.997018 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db626824-b4c5-42b9-91a6-d896e4523a0c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:18.998541 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db626824-b4c5-42b9-91a6-d896e4523a0c None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:19.023602 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:19.032407 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 40/80] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:43:18 2022] POST /identity/v3/users => generated 289 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:43:19.038326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffd2fe8-df16-4ac9-b789-5594c1f9740a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:19.044272 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db626824-b4c5-42b9-91a6-d896e4523a0c None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:19.085035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507591ae-7a64-4772-8779-06f357f3cf8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:19.086081 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507591ae-7a64-4772-8779-06f357f3cf8b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:19.086611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507591ae-7a64-4772-8779-06f357f3cf8b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:19.087184 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507591ae-7a64-4772-8779-06f357f3cf8b None admin] PATH_INFO: `/v3/users/620e6f7dab50405d9bebd872ceb59dc7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:19.088811 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507591ae-7a64-4772-8779-06f357f3cf8b None admin] RBAC: Authorizing `identity:get_user(user_id=620e6f7dab50405d9bebd872ceb59dc7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:19.091002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507591ae-7a64-4772-8779-06f357f3cf8b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:19.093222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 41/81] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:18 2022] GET /identity/v3/users/620e6f7dab50405d9bebd872ceb59dc7 => generated 275 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:19.094467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507591ae-7a64-4772-8779-06f357f3cf8b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:19.112458 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:19.200741 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db626824-b4c5-42b9-91a6-d896e4523a0c None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:19.222518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507591ae-7a64-4772-8779-06f357f3cf8b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:19.254924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8413105a-53be-44df-b1d0-daa074728492 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:19.255215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8413105a-53be-44df-b1d0-daa074728492 None admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:19.255469 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8413105a-53be-44df-b1d0-daa074728492 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:19.255752 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8413105a-53be-44df-b1d0-daa074728492 None admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2/users/620e6f7dab50405d9bebd872ceb59dc7/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:19.257896 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8413105a-53be-44df-b1d0-daa074728492 None admin] RBAC: Authorizing `identity:create_grant(project_id=c3343320b0624293946f3195f2d767e2, user_id=620e6f7dab50405d9bebd872ceb59dc7, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:19.259539 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8413105a-53be-44df-b1d0-daa074728492 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:19.275086 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12e02a1-b0d6-4d4b-93d0-202034a5d217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:19.276204 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12e02a1-b0d6-4d4b-93d0-202034a5d217 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:19.277295 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12e02a1-b0d6-4d4b-93d0-202034a5d217 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:19.277869 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12e02a1-b0d6-4d4b-93d0-202034a5d217 None admin] PATH_INFO: `/v3/projects/17e6fd7708d2409ca2451f4322aa43cc` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:19.279403 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12e02a1-b0d6-4d4b-93d0-202034a5d217 None admin] RBAC: Authorizing `identity:get_project(project_id=17e6fd7708d2409ca2451f4322aa43cc)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:19.281116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12e02a1-b0d6-4d4b-93d0-202034a5d217 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:19.283629 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 41/82] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:43:19 2022] PUT /identity/v3/projects/c3343320b0624293946f3195f2d767e2/users/620e6f7dab50405d9bebd872ceb59dc7/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:43:19.284448 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8413105a-53be-44df-b1d0-daa074728492 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:19.289286 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 42/83] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:19 2022] GET /identity/v3/projects/17e6fd7708d2409ca2451f4322aa43cc => generated 312 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:19.290846 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12e02a1-b0d6-4d4b-93d0-202034a5d217 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:19.402089 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:19.409940 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:19.435900 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12e02a1-b0d6-4d4b-93d0-202034a5d217 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:19.453510 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8413105a-53be-44df-b1d0-daa074728492 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:19.495816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b596a711-565a-44be-8230-ae78ce1a24f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:19.496815 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b596a711-565a-44be-8230-ae78ce1a24f9 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:19.497104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b596a711-565a-44be-8230-ae78ce1a24f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:19.497393 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b596a711-565a-44be-8230-ae78ce1a24f9 None admin] PATH_INFO: `/v3/projects/17e6fd7708d2409ca2451f4322aa43cc/users/620e6f7dab50405d9bebd872ceb59dc7/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:19.499919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b596a711-565a-44be-8230-ae78ce1a24f9 None admin] RBAC: Authorizing `identity:create_grant(project_id=17e6fd7708d2409ca2451f4322aa43cc, user_id=620e6f7dab50405d9bebd872ceb59dc7, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:19.501440 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b596a711-565a-44be-8230-ae78ce1a24f9 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:19.505943 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42057e05-5511-4180-aecb-c2be8060ba13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:19.507175 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42057e05-5511-4180-aecb-c2be8060ba13 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:19.507721 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42057e05-5511-4180-aecb-c2be8060ba13 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:19.508077 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42057e05-5511-4180-aecb-c2be8060ba13 None admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:19.509924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42057e05-5511-4180-aecb-c2be8060ba13 None admin] RBAC: Authorizing `identity:get_project(project_id=c3343320b0624293946f3195f2d767e2)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:19.512938 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42057e05-5511-4180-aecb-c2be8060ba13 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:19.516111 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 42/84] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:19 2022] GET /identity/v3/projects/c3343320b0624293946f3195f2d767e2 => generated 298 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:19.526100 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42057e05-5511-4180-aecb-c2be8060ba13 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:19.530345 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 43/85] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:43:19 2022] PUT /identity/v3/projects/17e6fd7708d2409ca2451f4322aa43cc/users/620e6f7dab50405d9bebd872ceb59dc7/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:43:19.735892 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42057e05-5511-4180-aecb-c2be8060ba13 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:19.796858 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:19.820398 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce23fce-9ccd-4231-890c-754cb616f476 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:19.821585 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce23fce-9ccd-4231-890c-754cb616f476 None admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:19.822181 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce23fce-9ccd-4231-890c-754cb616f476 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:19.822692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce23fce-9ccd-4231-890c-754cb616f476 None admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2/users/620e6f7dab50405d9bebd872ceb59dc7/roles/d9dcb4cfc0a245a98a7d96c100a497a1` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:19.828343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce23fce-9ccd-4231-890c-754cb616f476 None admin] RBAC: Authorizing `identity:create_grant(project_id=c3343320b0624293946f3195f2d767e2, user_id=620e6f7dab50405d9bebd872ceb59dc7, role_id=d9dcb4cfc0a245a98a7d96c100a497a1)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:19.831453 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce23fce-9ccd-4231-890c-754cb616f476 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:19.859900 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 43/86] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:43:19 2022] PUT /identity/v3/projects/c3343320b0624293946f3195f2d767e2/users/620e6f7dab50405d9bebd872ceb59dc7/roles/d9dcb4cfc0a245a98a7d96c100a497a1 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:43:22.279076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a5f6d5-4762-42c4-b472-b7717db71d91 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:22.280159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a5f6d5-4762-42c4-b472-b7717db71d91 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:22.280651 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a5f6d5-4762-42c4-b472-b7717db71d91 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:22.282708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 44/87] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:22 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:22.288563 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9778dd-d8e9-4b74-9b48-d98c39ecac43 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:22.289472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9778dd-d8e9-4b74-9b48-d98c39ecac43 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:22.289976 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9778dd-d8e9-4b74-9b48-d98c39ecac43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:22.320152 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-ce9778dd-d8e9-4b74-9b48-d98c39ecac43 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:22.321788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-ce9778dd-d8e9-4b74-9b48-d98c39ecac43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:22.410476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 44/88] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:22.565446 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5803082e-3b3b-431d-a020-52ea8e9365a5 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:22.566110 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5803082e-3b3b-431d-a020-52ea8e9365a5 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:22.566586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5803082e-3b3b-431d-a020-52ea8e9365a5 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:22.568271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 45/89] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:22.575136 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f10778d-c013-4fc9-be65-dc8dc6a6f537 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:22.575439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f10778d-c013-4fc9-be65-dc8dc6a6f537 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:22.575702 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f10778d-c013-4fc9-be65-dc8dc6a6f537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:22.603990 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-4f10778d-c013-4fc9-be65-dc8dc6a6f537 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:22.606242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4f10778d-c013-4fc9-be65-dc8dc6a6f537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:22.704468 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 45/90] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:22.968813 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5803082e-3b3b-431d-a020-52ea8e9365a5 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:22.980611 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35169b3a-bf81-475c-90c9-1d06197e1c5e None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:22.981272 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35169b3a-bf81-475c-90c9-1d06197e1c5e None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:22.981746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35169b3a-bf81-475c-90c9-1d06197e1c5e None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:22.983499 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 46/91] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:43:22.997286 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bab986-83d5-4eda-8b03-6a3b1e584eaa None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:22.997965 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bab986-83d5-4eda-8b03-6a3b1e584eaa None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:22.998437 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bab986-83d5-4eda-8b03-6a3b1e584eaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:23.030252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-a1bab986-83d5-4eda-8b03-6a3b1e584eaa None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:23.034137 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-a1bab986-83d5-4eda-8b03-6a3b1e584eaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:23.135594 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:23.208950 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 47/92] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:43:23.209981 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bab986-83d5-4eda-8b03-6a3b1e584eaa None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:23.210786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5803082e-3b3b-431d-a020-52ea8e9365a5 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:23.300508 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251eacd4-ca47-42a8-adfd-67aa15ea7d40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:23.301311 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251eacd4-ca47-42a8-adfd-67aa15ea7d40 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:23.301474 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251eacd4-ca47-42a8-adfd-67aa15ea7d40 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:23.301745 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251eacd4-ca47-42a8-adfd-67aa15ea7d40 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:23.325108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:23.334038 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251eacd4-ca47-42a8-adfd-67aa15ea7d40 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:23.342101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-251eacd4-ca47-42a8-adfd-67aa15ea7d40 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:43:23.342101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:23.342101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:23.342101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:23.342101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:23.342101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:23.342101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:23.342101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:23.342101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:23.342101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:23.342101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:23.342101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:23.342101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:23.342101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:23.342101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:23.342101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:23.343116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:23.343116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:23.343116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:23.343116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:23.343116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:23.343116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:23.343116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:23.343116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:23.343116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:23.343116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:23.343116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:23.343116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:23.343116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:23.343116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:23.343116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:23.343116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:23.344333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:23.344333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:23.344333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:23.344333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:23.344333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:23.344333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:23.344333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:23.344333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:23.344333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:23.344333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:23.344333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:43:23.344333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:23.344333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 46/93] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:43:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 375 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:23.346181 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a838f3-0adf-413a-ad96-cb38d413421f None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:23.346181 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a838f3-0adf-413a-ad96-cb38d413421f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:23.346504 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a838f3-0adf-413a-ad96-cb38d413421f None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:23.348762 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 47/94] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:23.357625 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb43c93-6d36-45e5-a936-971e1d68cd08 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:23.358082 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb43c93-6d36-45e5-a936-971e1d68cd08 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:23.358421 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb43c93-6d36-45e5-a936-971e1d68cd08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:23.382856 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-7bb43c93-6d36-45e5-a936-971e1d68cd08 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:23.384445 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7bb43c93-6d36-45e5-a936-971e1d68cd08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:23.443456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bab986-83d5-4eda-8b03-6a3b1e584eaa None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:23.488501 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029ad1e0-c741-4ea6-9f97-0d1cbefde22c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:23.489260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ad1e0-c741-4ea6-9f97-0d1cbefde22c None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:23.489589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ad1e0-c741-4ea6-9f97-0d1cbefde22c None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:23.489932 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ad1e0-c741-4ea6-9f97-0d1cbefde22c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:23.504942 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029ad1e0-c741-4ea6-9f97-0d1cbefde22c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:23.512010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-029ad1e0-c741-4ea6-9f97-0d1cbefde22c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 16 00:43:23.512010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:23.512010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:23.512010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:23.512010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:23.512010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:23.512010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:23.512010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:23.512010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:23.512010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:23.512010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:23.512010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:23.512010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:23.512010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:23.512010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:23.512010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:23.513628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:23.513628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:23.513628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:23.513628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:23.513628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:23.513628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:23.513628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:23.513628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:23.513628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:23.513628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:23.513628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:23.513628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:23.513628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:23.513628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:23.513628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:23.513628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:23.515089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:23.515089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:23.515089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:23.515089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:23.515089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:23.515089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:23.515089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:23.515089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:23.515089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:23.515089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:23.515089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 16 00:43:23.515089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:23.515089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 48/95] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:23 2022] GET /identity/v3/roles/reader => generated 84 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:23.515089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029ad1e0-c741-4ea6-9f97-0d1cbefde22c None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:23.518943 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 48/96] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:43:23.527693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dd5164-18ff-4ab6-a172-1de3942ac39c None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:23.528546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dd5164-18ff-4ab6-a172-1de3942ac39c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:23.529275 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dd5164-18ff-4ab6-a172-1de3942ac39c None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:23.531962 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 49/97] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:23 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:23.533735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dd5164-18ff-4ab6-a172-1de3942ac39c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:23.563670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:23.701712 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dd5164-18ff-4ab6-a172-1de3942ac39c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:23.702954 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029ad1e0-c741-4ea6-9f97-0d1cbefde22c None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:23.740994 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:23.748603 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c50f5f5-9f6b-45d8-b88d-d463e5693ff8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:23.749018 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03b2397-739c-4d6c-b110-7cd569ec2cc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:23.749310 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c50f5f5-9f6b-45d8-b88d-d463e5693ff8 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:23.749585 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c50f5f5-9f6b-45d8-b88d-d463e5693ff8 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:23.749731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03b2397-739c-4d6c-b110-7cd569ec2cc9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:23.749861 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c50f5f5-9f6b-45d8-b88d-d463e5693ff8 None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:23.749987 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03b2397-739c-4d6c-b110-7cd569ec2cc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:23.750111 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03b2397-739c-4d6c-b110-7cd569ec2cc9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:23.750460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c50f5f5-9f6b-45d8-b88d-d463e5693ff8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:23.750764 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c50f5f5-9f6b-45d8-b88d-d463e5693ff8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:23.752220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c50f5f5-9f6b-45d8-b88d-d463e5693ff8 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:23.764217 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 49/98] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:23.764404 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03b2397-739c-4d6c-b110-7cd569ec2cc9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:23.769741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-a03b2397-739c-4d6c-b110-7cd569ec2cc9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:23.769741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:23.769741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:23.769741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:23.769741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:23.769741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:23.769741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:23.769741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:23.769741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:23.769741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:23.769741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:23.769741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:23.769741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:23.769741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:23.769741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:23.769741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:23.770756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:23.770756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:23.770756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:23.770756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:23.770756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:23.770756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:23.770756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:23.770756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:23.770756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:23.770756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:23.770756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:23.770756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:23.770756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:23.770756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:23.770756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:23.770756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:23.771671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:23.771671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:23.771671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:23.771671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:23.771671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:23.771671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:23.771671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:23.771671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:23.771671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:23.771671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:23.771671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:23.771671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:23.772251 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c50f5f5-9f6b-45d8-b88d-d463e5693ff8 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:23.772321 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 50/99] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:23.774371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a232d741-ff8c-4f96-9c2a-bbb5499c2eff None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:23.775289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a232d741-ff8c-4f96-9c2a-bbb5499c2eff None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:23.775799 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a232d741-ff8c-4f96-9c2a-bbb5499c2eff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:23.811536 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a232d741-ff8c-4f96-9c2a-bbb5499c2eff None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:23.813820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a232d741-ff8c-4f96-9c2a-bbb5499c2eff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:23.930566 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 51/100] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:23.932170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462c55a0-e762-46c2-8e3e-54b0271b5914 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:23.932473 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462c55a0-e762-46c2-8e3e-54b0271b5914 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:23.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462c55a0-e762-46c2-8e3e-54b0271b5914 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:23.934965 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 52/101] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:23.935831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462c55a0-e762-46c2-8e3e-54b0271b5914 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:23.968112 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:23.976396 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c50f5f5-9f6b-45d8-b88d-d463e5693ff8 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:24.046013 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec3934c-e517-4e45-a11f-d64c4c5d09c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:24.047127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec3934c-e517-4e45-a11f-d64c4c5d09c7 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:24.047614 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec3934c-e517-4e45-a11f-d64c4c5d09c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:24.048090 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec3934c-e517-4e45-a11f-d64c4c5d09c7 None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:24.049292 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec3934c-e517-4e45-a11f-d64c4c5d09c7 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:24.049820 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec3934c-e517-4e45-a11f-d64c4c5d09c7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:24.052518 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec3934c-e517-4e45-a11f-d64c4c5d09c7 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:24.069040 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 50/102] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:43:23 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:24.070277 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec3934c-e517-4e45-a11f-d64c4c5d09c7 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:24.112279 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462c55a0-e762-46c2-8e3e-54b0271b5914 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:24.153615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:24.177397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c919ce41-4432-43b6-ade7-89194cda324d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:24.178654 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c919ce41-4432-43b6-ade7-89194cda324d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:24.179331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c919ce41-4432-43b6-ade7-89194cda324d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:24.179925 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c919ce41-4432-43b6-ade7-89194cda324d None admin] PATH_INFO: `/v3/users/68c5eeb505a240daa471059d6635deea` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:24.181853 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c919ce41-4432-43b6-ade7-89194cda324d None admin] RBAC: Authorizing `identity:get_user(user_id=68c5eeb505a240daa471059d6635deea)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:24.185277 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c919ce41-4432-43b6-ade7-89194cda324d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:24.188295 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 53/103] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:23 2022] GET /identity/v3/users/68c5eeb505a240daa471059d6635deea => generated 247 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:24.189609 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c919ce41-4432-43b6-ade7-89194cda324d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:24.248190 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec3934c-e517-4e45-a11f-d64c4c5d09c7 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:24.289566 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd37c26d-e69e-4a53-9693-346db5cc7d44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:24.290281 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd37c26d-e69e-4a53-9693-346db5cc7d44 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:24.290680 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd37c26d-e69e-4a53-9693-346db5cc7d44 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:24.291240 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd37c26d-e69e-4a53-9693-346db5cc7d44 None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:24.292056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd37c26d-e69e-4a53-9693-346db5cc7d44 None admin] RBAC: Adding query filter params (name=member) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:24.292575 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd37c26d-e69e-4a53-9693-346db5cc7d44 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:24.294045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd37c26d-e69e-4a53-9693-346db5cc7d44 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:24.307174 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 51/104] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:43:24 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:24.309893 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fdb2a5-6803-4ca7-89fd-1bc3f931f68a None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:24.310354 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fdb2a5-6803-4ca7-89fd-1bc3f931f68a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:24.310809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fdb2a5-6803-4ca7-89fd-1bc3f931f68a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:24.328703 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c919ce41-4432-43b6-ade7-89194cda324d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:24.331080 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-51fdb2a5-6803-4ca7-89fd-1bc3f931f68a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:24.332620 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-51fdb2a5-6803-4ca7-89fd-1bc3f931f68a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:24.353365 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:24.377188 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7ae107-a392-4cfa-93dd-ab9adff5e833 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:24.378186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7ae107-a392-4cfa-93dd-ab9adff5e833 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:24.378634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7ae107-a392-4cfa-93dd-ab9adff5e833 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:24.379120 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7ae107-a392-4cfa-93dd-ab9adff5e833 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:24.380893 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7ae107-a392-4cfa-93dd-ab9adff5e833 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:24.382995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7ae107-a392-4cfa-93dd-ab9adff5e833 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:24.386460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 54/105] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:43:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:24.388237 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7ae107-a392-4cfa-93dd-ab9adff5e833 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:24.437194 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 52/106] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:24.438590 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fdb2a5-6803-4ca7-89fd-1bc3f931f68a None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:24.485037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:24.565742 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fdb2a5-6803-4ca7-89fd-1bc3f931f68a None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:24.580948 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7ae107-a392-4cfa-93dd-ab9adff5e833 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:24.610101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985efd54-bff9-47ef-9ec4-becba7854a44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:24.611055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985efd54-bff9-47ef-9ec4-becba7854a44 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:24.611478 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985efd54-bff9-47ef-9ec4-becba7854a44 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:24.611878 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985efd54-bff9-47ef-9ec4-becba7854a44 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:24.642262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:24.647932 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b044c26f-fb4c-47a9-a650-4f1e630b563f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:24.648992 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b044c26f-fb4c-47a9-a650-4f1e630b563f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:24.649553 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b044c26f-fb4c-47a9-a650-4f1e630b563f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:24.650064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b044c26f-fb4c-47a9-a650-4f1e630b563f None admin] PATH_INFO: `/v3/users/9c4ab9e7c0564c809c3ead6dda4429d2` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:24.674661 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b044c26f-fb4c-47a9-a650-4f1e630b563f None admin] RBAC: Authorizing `identity:get_user(user_id=9c4ab9e7c0564c809c3ead6dda4429d2)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:24.677363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b044c26f-fb4c-47a9-a650-4f1e630b563f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:24.680527 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 55/107] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:24 2022] GET /identity/v3/users/9c4ab9e7c0564c809c3ead6dda4429d2 => generated 289 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:24.681702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b044c26f-fb4c-47a9-a650-4f1e630b563f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:24.729488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985efd54-bff9-47ef-9ec4-becba7854a44 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:24.732977 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985efd54-bff9-47ef-9ec4-becba7854a44 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:24.816845 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b044c26f-fb4c-47a9-a650-4f1e630b563f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:24.837544 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:24.870680 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0809721-be3f-4a0c-91eb-bd3ec0002fab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:24.871708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0809721-be3f-4a0c-91eb-bd3ec0002fab None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:24.872058 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0809721-be3f-4a0c-91eb-bd3ec0002fab None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:24.872399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0809721-be3f-4a0c-91eb-bd3ec0002fab None admin] PATH_INFO: `/v3/roles/member` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:24.884605 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0809721-be3f-4a0c-91eb-bd3ec0002fab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:24.890182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-a0809721-be3f-4a0c-91eb-bd3ec0002fab None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:24.890182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:24.890182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:24.890182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:24.890182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:24.890182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:24.890182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:24.890182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:24.890182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:24.890182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:24.890182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:24.890182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:24.890182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:24.890182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:24.890182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:24.890182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:24.891930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:24.891930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:24.891930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:24.891930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:24.891930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:24.891930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:24.891930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:24.891930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:24.891930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:24.891930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:24.891930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:24.891930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:24.891930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:24.891930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:24.891930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:24.891930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:24.893200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:24.893200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:24.893200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:24.893200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:24.893200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:24.893200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:24.893200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:24.893200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:24.893200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:24.893200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:24.893200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:24.893200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:24.893200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 56/109] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:24.893914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 53/109] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:43:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 454 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:43:24.893984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0809721-be3f-4a0c-91eb-bd3ec0002fab None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:24.894057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985efd54-bff9-47ef-9ec4-becba7854a44 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:24.976462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:25.034513 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0809721-be3f-4a0c-91eb-bd3ec0002fab None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:25.048069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985efd54-bff9-47ef-9ec4-becba7854a44 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:25.080780 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2865e4c4-af06-4d41-a217-3f48ff14b3f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:25.081597 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2865e4c4-af06-4d41-a217-3f48ff14b3f9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:25.081939 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2865e4c4-af06-4d41-a217-3f48ff14b3f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:25.082367 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2865e4c4-af06-4d41-a217-3f48ff14b3f9 None admin] PATH_INFO: `/v3/users/620e6f7dab50405d9bebd872ceb59dc7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:25.083976 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2865e4c4-af06-4d41-a217-3f48ff14b3f9 None admin] RBAC: Authorizing `identity:get_user(user_id=620e6f7dab50405d9bebd872ceb59dc7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:25.085676 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2865e4c4-af06-4d41-a217-3f48ff14b3f9 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:25.087617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 57/110] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:24 2022] GET /identity/v3/users/620e6f7dab50405d9bebd872ceb59dc7 => generated 275 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:25.088573 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2865e4c4-af06-4d41-a217-3f48ff14b3f9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:25.096585 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749f2098-19f7-4558-adae-086225dce4e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:25.097461 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749f2098-19f7-4558-adae-086225dce4e2 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:25.097786 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749f2098-19f7-4558-adae-086225dce4e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:25.098099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749f2098-19f7-4558-adae-086225dce4e2 None admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:25.099121 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749f2098-19f7-4558-adae-086225dce4e2 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:25.101639 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749f2098-19f7-4558-adae-086225dce4e2 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:25.142092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 54/111] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 16 00:43:24 2022] POST /identity/v3/projects => generated 302 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:43:25.154250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749f2098-19f7-4558-adae-086225dce4e2 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:25.162156 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:25.255087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2865e4c4-af06-4d41-a217-3f48ff14b3f9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:25.306175 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b58ddd-f964-4bb9-8d0f-5013df92d650 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:25.307370 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b58ddd-f964-4bb9-8d0f-5013df92d650 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:25.308392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b58ddd-f964-4bb9-8d0f-5013df92d650 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:25.308847 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b58ddd-f964-4bb9-8d0f-5013df92d650 None admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:25.311216 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b58ddd-f964-4bb9-8d0f-5013df92d650 None admin] RBAC: Authorizing `identity:get_project(project_id=c3343320b0624293946f3195f2d767e2)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:25.314222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b58ddd-f964-4bb9-8d0f-5013df92d650 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:25.322720 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 58/112] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:25 2022] GET /identity/v3/projects/c3343320b0624293946f3195f2d767e2 => generated 298 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:25.322720 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b58ddd-f964-4bb9-8d0f-5013df92d650 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:25.353888 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749f2098-19f7-4558-adae-086225dce4e2 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:25.358118 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:25.409772 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d1c7df-cc63-4c42-9331-420d2aab37b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:25.410919 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d1c7df-cc63-4c42-9331-420d2aab37b3 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:25.411427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d1c7df-cc63-4c42-9331-420d2aab37b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:25.412118 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d1c7df-cc63-4c42-9331-420d2aab37b3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:25.432741 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d1c7df-cc63-4c42-9331-420d2aab37b3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:25.442491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-16d1c7df-cc63-4c42-9331-420d2aab37b3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:43:25.442491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:25.442491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:25.442491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:25.442491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:25.442491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:25.442491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:25.442491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:25.442491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:25.442491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:25.442491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:25.442491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:25.442491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:25.442491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:25.442491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:25.442491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:25.443729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:25.443729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:25.443729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:25.443729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:25.443729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:25.443729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:25.443729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:25.443729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:25.443729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:25.443729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:25.443729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:25.443729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:25.443729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:25.443729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:25.443729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:25.443729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:25.444704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:25.444704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:25.444704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:25.444704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:25.444704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:25.444704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:25.444704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:25.444704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:25.444704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:25.444704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:25.444704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:43:25.444704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:25.444704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 55/113] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 16 00:43:25 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:25.445767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d1c7df-cc63-4c42-9331-420d2aab37b3 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:25.470650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b58ddd-f964-4bb9-8d0f-5013df92d650 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:25.484429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:25.512419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f99baa9-918c-41d3-aa0a-5f1a95c5e9a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:25.513063 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f99baa9-918c-41d3-aa0a-5f1a95c5e9a3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:25.513353 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f99baa9-918c-41d3-aa0a-5f1a95c5e9a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:25.513629 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f99baa9-918c-41d3-aa0a-5f1a95c5e9a3 None admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:25.514773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f99baa9-918c-41d3-aa0a-5f1a95c5e9a3 None admin] RBAC: Authorizing `identity:get_project(project_id=c3343320b0624293946f3195f2d767e2)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:25.516290 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f99baa9-918c-41d3-aa0a-5f1a95c5e9a3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:25.522905 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 59/114] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:25 2022] GET /identity/v3/projects/c3343320b0624293946f3195f2d767e2 => generated 298 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:25.522905 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f99baa9-918c-41d3-aa0a-5f1a95c5e9a3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:25.595450 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d1c7df-cc63-4c42-9331-420d2aab37b3 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:25.640837 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8c8239-560d-49b0-afe5-07bdf9175ead None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:25.641502 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8c8239-560d-49b0-afe5-07bdf9175ead None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:25.641877 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8c8239-560d-49b0-afe5-07bdf9175ead None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:25.643004 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8c8239-560d-49b0-afe5-07bdf9175ead None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:25.643837 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8c8239-560d-49b0-afe5-07bdf9175ead None admin] RBAC: Adding query filter params (name=member) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:25.644220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8c8239-560d-49b0-afe5-07bdf9175ead None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:25.646141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8c8239-560d-49b0-afe5-07bdf9175ead None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:25.661508 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 56/115] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:43:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:25.661508 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8c8239-560d-49b0-afe5-07bdf9175ead None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:25.668765 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f99baa9-918c-41d3-aa0a-5f1a95c5e9a3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:25.691991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:25.723937 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d65dbd-6d33-4faf-a9d7-796128ebe394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:25.724566 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d65dbd-6d33-4faf-a9d7-796128ebe394 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:25.724864 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d65dbd-6d33-4faf-a9d7-796128ebe394 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:25.725141 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d65dbd-6d33-4faf-a9d7-796128ebe394 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:25.779131 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:25.804877 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d65dbd-6d33-4faf-a9d7-796128ebe394 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:25.806417 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d65dbd-6d33-4faf-a9d7-796128ebe394 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:25.819864 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8c8239-560d-49b0-afe5-07bdf9175ead None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:25.859101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e9cca3-d50b-4ac9-9533-f040201c6592 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:25.860027 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e9cca3-d50b-4ac9-9533-f040201c6592 None admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:25.860476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e9cca3-d50b-4ac9-9533-f040201c6592 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:25.860894 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e9cca3-d50b-4ac9-9533-f040201c6592 None admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2/users/9c4ab9e7c0564c809c3ead6dda4429d2/roles/1ac2f92c6eb34313ac9d9dfd4476663c` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:25.863023 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e9cca3-d50b-4ac9-9533-f040201c6592 None admin] RBAC: Authorizing `identity:create_grant(project_id=c3343320b0624293946f3195f2d767e2, user_id=9c4ab9e7c0564c809c3ead6dda4429d2, role_id=1ac2f92c6eb34313ac9d9dfd4476663c)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:25.864626 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e9cca3-d50b-4ac9-9533-f040201c6592 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:25.890984 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 57/116] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:43:25 2022] PUT /identity/v3/projects/c3343320b0624293946f3195f2d767e2/users/9c4ab9e7c0564c809c3ead6dda4429d2/roles/1ac2f92c6eb34313ac9d9dfd4476663c => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:43:25.890984 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e9cca3-d50b-4ac9-9533-f040201c6592 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:25.988864 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 60/117] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:43:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 470 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:43:25.997627 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d65dbd-6d33-4faf-a9d7-796128ebe394 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:26.078744 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:26.116094 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e9cca3-d50b-4ac9-9533-f040201c6592 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:26.176523 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:26.189819 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3688e9-c338-41b4-81b4-2be5051f9cdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:26.190913 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3688e9-c338-41b4-81b4-2be5051f9cdb None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:26.192026 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3688e9-c338-41b4-81b4-2be5051f9cdb None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:26.192539 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3688e9-c338-41b4-81b4-2be5051f9cdb None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:26.193794 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3688e9-c338-41b4-81b4-2be5051f9cdb None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:26.194346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3688e9-c338-41b4-81b4-2be5051f9cdb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:26.197043 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3688e9-c338-41b4-81b4-2be5051f9cdb None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:26.217128 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 58/118] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:43:25 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 336 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:26.218629 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3688e9-c338-41b4-81b4-2be5051f9cdb None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:26.227588 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d65dbd-6d33-4faf-a9d7-796128ebe394 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:26.291051 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f331a06-d751-437f-9e71-ab64f0c0d3c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:26.292737 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f331a06-d751-437f-9e71-ab64f0c0d3c1 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:26.293320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f331a06-d751-437f-9e71-ab64f0c0d3c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:26.293910 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f331a06-d751-437f-9e71-ab64f0c0d3c1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:26.295450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f331a06-d751-437f-9e71-ab64f0c0d3c1 None admin] RBAC: Adding query filter params (role.id=c5cd777d70a649028f970fd986dc484f, scope.project.id=c3343320b0624293946f3195f2d767e2, user.id=68c5eeb505a240daa471059d6635deea) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:26.296099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f331a06-d751-437f-9e71-ab64f0c0d3c1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:26.299016 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f331a06-d751-437f-9e71-ab64f0c0d3c1 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:26.313729 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 61/119] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:43:25 2022] GET /identity/v3/role_assignments?user.id=68c5eeb505a240daa471059d6635deea&scope.project.id=c3343320b0624293946f3195f2d767e2&role.id=c5cd777d70a649028f970fd986dc484f => generated 608 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:26.315151 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f331a06-d751-437f-9e71-ab64f0c0d3c1 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:26.502981 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3688e9-c338-41b4-81b4-2be5051f9cdb None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:26.527068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f331a06-d751-437f-9e71-ab64f0c0d3c1 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:26.629964 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a08287c-e183-4095-a217-d10e5145e47a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:26.631126 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a08287c-e183-4095-a217-d10e5145e47a None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:26.631638 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a08287c-e183-4095-a217-d10e5145e47a None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:26.632145 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a08287c-e183-4095-a217-d10e5145e47a None admin] PATH_INFO: `/v3/users/620e6f7dab50405d9bebd872ceb59dc7` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:26.634419 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a08287c-e183-4095-a217-d10e5145e47a None admin] RBAC: Authorizing `identity:get_user(user_id=620e6f7dab50405d9bebd872ceb59dc7)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:26.639272 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a08287c-e183-4095-a217-d10e5145e47a None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:26.640625 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4866b037-9f8e-4a18-851d-dc9173523f89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:26.641469 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 59/120] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:26 2022] GET /identity/v3/users/620e6f7dab50405d9bebd872ceb59dc7 => generated 275 bytes in 424 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:26.642455 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a08287c-e183-4095-a217-d10e5145e47a None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:26.643037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4866b037-9f8e-4a18-851d-dc9173523f89 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:26.647230 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4866b037-9f8e-4a18-851d-dc9173523f89 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:26.648369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4866b037-9f8e-4a18-851d-dc9173523f89 None admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:26.652646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4866b037-9f8e-4a18-851d-dc9173523f89 None admin] RBAC: Authorizing `identity:get_project(project_id=c3343320b0624293946f3195f2d767e2)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:26.656568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4866b037-9f8e-4a18-851d-dc9173523f89 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:26.661503 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 62/121] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:26 2022] GET /identity/v3/projects/c3343320b0624293946f3195f2d767e2 => generated 298 bytes in 347 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:26.663347 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4866b037-9f8e-4a18-851d-dc9173523f89 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:26.673543 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:26.696097 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:26.775520 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a08287c-e183-4095-a217-d10e5145e47a None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:26.821028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e4575a-8bd0-4dc6-a7db-00235d08b962 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:26.821723 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e4575a-8bd0-4dc6-a7db-00235d08b962 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:26.822009 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e4575a-8bd0-4dc6-a7db-00235d08b962 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:26.822285 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e4575a-8bd0-4dc6-a7db-00235d08b962 None admin] PATH_INFO: `/v3/users/620e6f7dab50405d9bebd872ceb59dc7` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:26.823515 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e4575a-8bd0-4dc6-a7db-00235d08b962 None admin] RBAC: Authorizing `identity:get_user(user_id=620e6f7dab50405d9bebd872ceb59dc7)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:26.824981 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e4575a-8bd0-4dc6-a7db-00235d08b962 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:26.826992 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 60/122] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:26 2022] GET /identity/v3/users/620e6f7dab50405d9bebd872ceb59dc7 => generated 275 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:26.827886 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e4575a-8bd0-4dc6-a7db-00235d08b962 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:26.886955 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4866b037-9f8e-4a18-851d-dc9173523f89 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:26.943282 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:26.952120 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81091de3-5f6c-4657-ab33-8a1af77be774 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:26.953675 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81091de3-5f6c-4657-ab33-8a1af77be774 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:26.954459 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81091de3-5f6c-4657-ab33-8a1af77be774 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:26.955485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81091de3-5f6c-4657-ab33-8a1af77be774 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:26.963295 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e4575a-8bd0-4dc6-a7db-00235d08b962 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:27.008143 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3fc4bd-79ad-4a23-a1e8-2deecc97c071 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:27.008700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3fc4bd-79ad-4a23-a1e8-2deecc97c071 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:27.009028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3fc4bd-79ad-4a23-a1e8-2deecc97c071 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:27.009312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3fc4bd-79ad-4a23-a1e8-2deecc97c071 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:27.009985 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3fc4bd-79ad-4a23-a1e8-2deecc97c071 None admin] RBAC: Adding query filter params (role.id=8a6c7a6c7d154f3db6d2639609d60224, scope.project.id=c3343320b0624293946f3195f2d767e2, user.id=620e6f7dab50405d9bebd872ceb59dc7) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:27.010300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3fc4bd-79ad-4a23-a1e8-2deecc97c071 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:27.012728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3fc4bd-79ad-4a23-a1e8-2deecc97c071 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:27.020305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 61/123] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:43:26 2022] GET /identity/v3/role_assignments?user.id=620e6f7dab50405d9bebd872ceb59dc7&scope.project.id=c3343320b0624293946f3195f2d767e2&role.id=8a6c7a6c7d154f3db6d2639609d60224 => generated 608 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:27.021363 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3fc4bd-79ad-4a23-a1e8-2deecc97c071 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:27.101319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81091de3-5f6c-4657-ab33-8a1af77be774 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:27.105367 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81091de3-5f6c-4657-ab33-8a1af77be774 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:27.139402 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:27.148400 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3fc4bd-79ad-4a23-a1e8-2deecc97c071 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:27.191033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036ff4cd-79f3-42b1-a370-26a6c65f48f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:27.191668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036ff4cd-79f3-42b1-a370-26a6c65f48f0 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:27.191950 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036ff4cd-79f3-42b1-a370-26a6c65f48f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:27.192226 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036ff4cd-79f3-42b1-a370-26a6c65f48f0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:27.223045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:27.286899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036ff4cd-79f3-42b1-a370-26a6c65f48f0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:27.288452 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036ff4cd-79f3-42b1-a370-26a6c65f48f0 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:27.415114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 63/124] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:43:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 752 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:43:27.422219 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81091de3-5f6c-4657-ab33-8a1af77be774 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:27.450739 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 62/125] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:43:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:43:27.460613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036ff4cd-79f3-42b1-a370-26a6c65f48f0 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:27.547174 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:27.623731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81091de3-5f6c-4657-ab33-8a1af77be774 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:27.646939 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036ff4cd-79f3-42b1-a370-26a6c65f48f0 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:27.657507 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:27.700521 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6464092c-0845-4349-8e22-709a51b3d3f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:27.701607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6464092c-0845-4349-8e22-709a51b3d3f9 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:27.702166 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6464092c-0845-4349-8e22-709a51b3d3f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:27.702598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6464092c-0845-4349-8e22-709a51b3d3f9 None admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:27.705106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6464092c-0845-4349-8e22-709a51b3d3f9 None admin] RBAC: Authorizing `identity:get_project(project_id=c3343320b0624293946f3195f2d767e2)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:27.705825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c31938-303d-461d-9b2a-3e3ab19925a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:27.707239 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c31938-303d-461d-9b2a-3e3ab19925a3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:27.708018 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6464092c-0845-4349-8e22-709a51b3d3f9 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:27.708209 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c31938-303d-461d-9b2a-3e3ab19925a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:27.708916 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c31938-303d-461d-9b2a-3e3ab19925a3 None admin] PATH_INFO: `/v3/projects/17e6fd7708d2409ca2451f4322aa43cc` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:27.711402 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 63/126] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:27 2022] GET /identity/v3/projects/c3343320b0624293946f3195f2d767e2 => generated 298 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:27.711724 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c31938-303d-461d-9b2a-3e3ab19925a3 None admin] RBAC: Authorizing `identity:get_project(project_id=17e6fd7708d2409ca2451f4322aa43cc)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:27.715164 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c31938-303d-461d-9b2a-3e3ab19925a3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:27.718927 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 64/127] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:27 2022] GET /identity/v3/projects/17e6fd7708d2409ca2451f4322aa43cc => generated 312 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:27.722381 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6464092c-0845-4349-8e22-709a51b3d3f9 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:27.726058 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c31938-303d-461d-9b2a-3e3ab19925a3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:27.903124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6464092c-0845-4349-8e22-709a51b3d3f9 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:27.928285 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c31938-303d-461d-9b2a-3e3ab19925a3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:27.936309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:27.968413 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a951741c-c6d8-4f22-8a4c-b33967af9e22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:27.969498 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a951741c-c6d8-4f22-8a4c-b33967af9e22 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:27.969955 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a951741c-c6d8-4f22-8a4c-b33967af9e22 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:27.970396 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a951741c-c6d8-4f22-8a4c-b33967af9e22 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:27.971704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a951741c-c6d8-4f22-8a4c-b33967af9e22 None admin] RBAC: Adding query filter params (role.id=d9dcb4cfc0a245a98a7d96c100a497a1, scope.project.id=c3343320b0624293946f3195f2d767e2, user.id=620e6f7dab50405d9bebd872ceb59dc7) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:27.972348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a951741c-c6d8-4f22-8a4c-b33967af9e22 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:27.974278 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a951741c-c6d8-4f22-8a4c-b33967af9e22 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:27.985508 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 64/128] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:43:27 2022] GET /identity/v3/role_assignments?user.id=620e6f7dab50405d9bebd872ceb59dc7&scope.project.id=c3343320b0624293946f3195f2d767e2&role.id=d9dcb4cfc0a245a98a7d96c100a497a1 => generated 608 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:27.987612 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ea2843-a324-46ba-98a4-70b1a1d6d34e None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:27.988114 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ea2843-a324-46ba-98a4-70b1a1d6d34e None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:27.988633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ea2843-a324-46ba-98a4-70b1a1d6d34e None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:27.990928 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 65/129] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:27 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:28.003127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fe31ed-5000-4147-a0b2-90672c2bea48 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:28.003127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fe31ed-5000-4147-a0b2-90672c2bea48 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:28.003127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fe31ed-5000-4147-a0b2-90672c2bea48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:28.004411 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6451f8e-f2dd-48d7-a18b-e4268ea324a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:28.005500 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6451f8e-f2dd-48d7-a18b-e4268ea324a3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:28.005669 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6451f8e-f2dd-48d7-a18b-e4268ea324a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:28.006165 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6451f8e-f2dd-48d7-a18b-e4268ea324a3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:28.007360 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6451f8e-f2dd-48d7-a18b-e4268ea324a3 None admin] RBAC: Adding query filter params (role.id=8a6c7a6c7d154f3db6d2639609d60224, scope.project.id=17e6fd7708d2409ca2451f4322aa43cc, user.id=620e6f7dab50405d9bebd872ceb59dc7) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:28.007872 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6451f8e-f2dd-48d7-a18b-e4268ea324a3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:28.010346 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6451f8e-f2dd-48d7-a18b-e4268ea324a3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:28.021418 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 65/130] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:43:27 2022] GET /identity/v3/role_assignments?user.id=620e6f7dab50405d9bebd872ceb59dc7&scope.project.id=17e6fd7708d2409ca2451f4322aa43cc&role.id=8a6c7a6c7d154f3db6d2639609d60224 => generated 608 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:28.033171 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-23fe31ed-5000-4147-a0b2-90672c2bea48 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:28.035418 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-23fe31ed-5000-4147-a0b2-90672c2bea48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:28.171890 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 66/131] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:43:28.567602 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6451f8e-f2dd-48d7-a18b-e4268ea324a3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:28.621767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:28.685263 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6451f8e-f2dd-48d7-a18b-e4268ea324a3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:28.723780 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8193fc-8101-4207-b457-5c6b81ef124d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:28.724430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8193fc-8101-4207-b457-5c6b81ef124d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:28.724711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8193fc-8101-4207-b457-5c6b81ef124d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:28.724983 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8193fc-8101-4207-b457-5c6b81ef124d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:28.726146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8193fc-8101-4207-b457-5c6b81ef124d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:28.727639 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8193fc-8101-4207-b457-5c6b81ef124d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:28.729242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 66/132] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:43:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:28.735585 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fe31ed-5000-4147-a0b2-90672c2bea48 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:28.853257 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fe31ed-5000-4147-a0b2-90672c2bea48 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:28.867492 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:28.896054 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5cba0fc-2c2d-4806-8d6a-6a19cb365e19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:28.896684 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cba0fc-2c2d-4806-8d6a-6a19cb365e19 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:28.896968 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cba0fc-2c2d-4806-8d6a-6a19cb365e19 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:28.897243 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cba0fc-2c2d-4806-8d6a-6a19cb365e19 None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:28.897918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5cba0fc-2c2d-4806-8d6a-6a19cb365e19 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:28.899444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5cba0fc-2c2d-4806-8d6a-6a19cb365e19 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:28.926844 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 67/133] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:43:28 2022] POST /identity/v3/users => generated 283 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:43:28.983011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16f0652-a8c2-42f4-ac28-a67579589394 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:28.983220 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16f0652-a8c2-42f4-ac28-a67579589394 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:28.983484 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16f0652-a8c2-42f4-ac28-a67579589394 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:28.985087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 67/134] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:28.990770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f072531c-05de-4412-a9e9-ab0e831a4b6e None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:28.991099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f072531c-05de-4412-a9e9-ab0e831a4b6e None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:28.991386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f072531c-05de-4412-a9e9-ab0e831a4b6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:29.012374 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-f072531c-05de-4412-a9e9-ab0e831a4b6e None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:29.013626 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-f072531c-05de-4412-a9e9-ab0e831a4b6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:29.094813 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 68/135] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:29.379770 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16f0652-a8c2-42f4-ac28-a67579589394 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:29.519005 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16f0652-a8c2-42f4-ac28-a67579589394 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:29.541172 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:29.585283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e1582f-f5a5-4522-b14c-2c6c778ea2ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:29.585975 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e1582f-f5a5-4522-b14c-2c6c778ea2ae None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:29.586268 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e1582f-f5a5-4522-b14c-2c6c778ea2ae None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:29.586550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e1582f-f5a5-4522-b14c-2c6c778ea2ae None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:29.601221 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e1582f-f5a5-4522-b14c-2c6c778ea2ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:29.606571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-a6e1582f-f5a5-4522-b14c-2c6c778ea2ae None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 16 00:43:29.606571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:29.606571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:29.606571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:29.606571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:29.606571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:29.606571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:29.606571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:29.606571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:29.606571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:29.606571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:29.606571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:29.606571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:29.606571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:29.606571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:29.606571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:29.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:29.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:29.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:29.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:29.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:29.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:29.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:29.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:29.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:29.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:29.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:29.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:29.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:29.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:29.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:29.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:29.608432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:29.608432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:29.608432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:29.608432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:29.608432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:29.608432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:29.608432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:29.608432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:29.608432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:29.608432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:29.608432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 16 00:43:29.608432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:29.608432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 68/136] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:29 2022] GET /identity/v3/roles/reader => generated 84 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:29.614229 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f072531c-05de-4412-a9e9-ab0e831a4b6e None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:29.727748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:29.735915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f072531c-05de-4412-a9e9-ab0e831a4b6e None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:29.783458 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce5c861-f0ac-41aa-8091-f2786016bc33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:29.784140 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce5c861-f0ac-41aa-8091-f2786016bc33 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:29.784425 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce5c861-f0ac-41aa-8091-f2786016bc33 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:29.784700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce5c861-f0ac-41aa-8091-f2786016bc33 None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:29.785398 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce5c861-f0ac-41aa-8091-f2786016bc33 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:29.785809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce5c861-f0ac-41aa-8091-f2786016bc33 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:29.787319 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce5c861-f0ac-41aa-8091-f2786016bc33 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:29.797109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 69/137] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:43:29 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:29.803455 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e1582f-f5a5-4522-b14c-2c6c778ea2ae None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:29.921984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e1582f-f5a5-4522-b14c-2c6c778ea2ae None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:29.962152 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb39dd4f-39c5-476c-a1c4-89d1fb963557 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:29.962839 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb39dd4f-39c5-476c-a1c4-89d1fb963557 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:29.963169 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb39dd4f-39c5-476c-a1c4-89d1fb963557 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:29.963823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb39dd4f-39c5-476c-a1c4-89d1fb963557 None admin] PATH_INFO: `/v3/users/9c4ab9e7c0564c809c3ead6dda4429d2` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:29.965275 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb39dd4f-39c5-476c-a1c4-89d1fb963557 None admin] RBAC: Authorizing `identity:get_user(user_id=9c4ab9e7c0564c809c3ead6dda4429d2)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:29.966833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb39dd4f-39c5-476c-a1c4-89d1fb963557 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:29.968985 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 69/138] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:29 2022] GET /identity/v3/users/9c4ab9e7c0564c809c3ead6dda4429d2 => generated 289 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:29.977717 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce5c861-f0ac-41aa-8091-f2786016bc33 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:30.183607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce5c861-f0ac-41aa-8091-f2786016bc33 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:30.255824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3337fbe2-1350-4189-adee-a5a65d081e01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:30.256858 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3337fbe2-1350-4189-adee-a5a65d081e01 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:30.257307 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3337fbe2-1350-4189-adee-a5a65d081e01 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:30.257751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3337fbe2-1350-4189-adee-a5a65d081e01 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:30.315375 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:30.399728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3337fbe2-1350-4189-adee-a5a65d081e01 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:30.402735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3337fbe2-1350-4189-adee-a5a65d081e01 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:30.594186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 70/139] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:43:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 618 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:43:30.600594 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb39dd4f-39c5-476c-a1c4-89d1fb963557 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:30.646229 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:30.719068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb39dd4f-39c5-476c-a1c4-89d1fb963557 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:30.758306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726b9df4-8981-4dec-9288-d75634267587 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:30.758997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726b9df4-8981-4dec-9288-d75634267587 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:30.759290 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726b9df4-8981-4dec-9288-d75634267587 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:30.759572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726b9df4-8981-4dec-9288-d75634267587 None admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:30.761330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726b9df4-8981-4dec-9288-d75634267587 None admin] RBAC: Authorizing `identity:get_project(project_id=c3343320b0624293946f3195f2d767e2)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:30.763188 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726b9df4-8981-4dec-9288-d75634267587 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:30.765407 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 70/140] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:30 2022] GET /identity/v3/projects/c3343320b0624293946f3195f2d767e2 => generated 298 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:30.772213 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3337fbe2-1350-4189-adee-a5a65d081e01 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:30.883357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:30.961476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3337fbe2-1350-4189-adee-a5a65d081e01 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:31.025531 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff672e0-5602-418f-8b8b-21a4a71d069a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:31.027269 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff672e0-5602-418f-8b8b-21a4a71d069a None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:31.027438 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff672e0-5602-418f-8b8b-21a4a71d069a None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:31.027909 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff672e0-5602-418f-8b8b-21a4a71d069a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:31.029409 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff672e0-5602-418f-8b8b-21a4a71d069a None admin] RBAC: Adding query filter params (role.id=1ac2f92c6eb34313ac9d9dfd4476663c, scope.project.id=c3343320b0624293946f3195f2d767e2, user.id=9c4ab9e7c0564c809c3ead6dda4429d2) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:31.029946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff672e0-5602-418f-8b8b-21a4a71d069a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:31.032959 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff672e0-5602-418f-8b8b-21a4a71d069a None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:31.045998 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 71/141] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:43:30 2022] GET /identity/v3/role_assignments?user.id=9c4ab9e7c0564c809c3ead6dda4429d2&scope.project.id=c3343320b0624293946f3195f2d767e2&role.id=1ac2f92c6eb34313ac9d9dfd4476663c => generated 608 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:31.719673 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feda60a7-04a0-4cf2-8fb0-71fbc30748e1 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:31.720384 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feda60a7-04a0-4cf2-8fb0-71fbc30748e1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:31.720552 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feda60a7-04a0-4cf2-8fb0-71fbc30748e1 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:31.722993 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 71/142] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:31 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:31.729170 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eac334-3101-4b2f-bc7d-79654328b3d6 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:31.730453 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eac334-3101-4b2f-bc7d-79654328b3d6 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:31.730955 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eac334-3101-4b2f-bc7d-79654328b3d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:31.755095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-73eac334-3101-4b2f-bc7d-79654328b3d6 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:31.756338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-73eac334-3101-4b2f-bc7d-79654328b3d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:31.831728 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b13eed1-a96b-481f-b1a6-28c76f2d6967 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:31.832267 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b13eed1-a96b-481f-b1a6-28c76f2d6967 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:31.832811 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b13eed1-a96b-481f-b1a6-28c76f2d6967 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:31.834324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 72/143] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:43:31.837096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 72/144] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:43:31.840839 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89152005-d787-4d6d-8eb7-78692cc6321e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:31.841332 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89152005-d787-4d6d-8eb7-78692cc6321e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:31.841788 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89152005-d787-4d6d-8eb7-78692cc6321e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:31.861221 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-89152005-d787-4d6d-8eb7-78692cc6321e None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:31.862694 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-89152005-d787-4d6d-8eb7-78692cc6321e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:31.941913 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 73/145] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:43:32.182778 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73eac334-3101-4b2f-bc7d-79654328b3d6 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:32.216289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89152005-d787-4d6d-8eb7-78692cc6321e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:32.257316 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:32.267303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:32.343086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73eac334-3101-4b2f-bc7d-79654328b3d6 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:32.344378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89152005-d787-4d6d-8eb7-78692cc6321e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:32.382241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2073165a-2b53-4a36-93d7-ca8bc5ee5260 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:32.382911 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2073165a-2b53-4a36-93d7-ca8bc5ee5260 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:32.383195 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2073165a-2b53-4a36-93d7-ca8bc5ee5260 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:32.383467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2073165a-2b53-4a36-93d7-ca8bc5ee5260 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:32.385047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2073165a-2b53-4a36-93d7-ca8bc5ee5260 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:32.388329 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2073165a-2b53-4a36-93d7-ca8bc5ee5260 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:32.390673 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 73/146] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:43:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:32.392560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8127247-f299-45cd-a80a-4b28178742b9 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:32.392846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8127247-f299-45cd-a80a-4b28178742b9 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:32.393120 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8127247-f299-45cd-a80a-4b28178742b9 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:32.393731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d045c0-e481-4b6d-8525-6d05c0f95b75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:32.394397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d045c0-e481-4b6d-8525-6d05c0f95b75 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:32.394507 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 74/147] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:32.394695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d045c0-e481-4b6d-8525-6d05c0f95b75 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:32.396127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8127247-f299-45cd-a80a-4b28178742b9 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:32.399569 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d045c0-e481-4b6d-8525-6d05c0f95b75 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:32.412923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d045c0-e481-4b6d-8525-6d05c0f95b75 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:32.419248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-10d045c0-e481-4b6d-8525-6d05c0f95b75 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:43:32.419248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:32.419248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:32.419248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:32.419248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:32.419248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:32.419248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:32.419248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:32.419248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:32.419248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:32.419248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:32.419248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:32.419248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:32.419248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:32.419248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:32.419248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:32.420303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:32.420303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:32.420303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:32.420303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:32.420303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:32.420303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:32.420303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:32.420303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:32.420303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:32.420303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:32.420303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:32.420303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:32.420303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:32.420303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:32.420303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:32.420303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:32.421598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:32.421598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:32.421598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:32.421598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:32.421598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:32.421598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:32.421598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:32.421598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:32.421598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:32.421598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:32.421598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:43:32.421598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:32.421598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 74/148] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 16 00:43:32 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:43:32.422333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c934121-d5b1-4fa0-b5c2-7b804bae1c34 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:32.422333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c934121-d5b1-4fa0-b5c2-7b804bae1c34 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:32.422456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c934121-d5b1-4fa0-b5c2-7b804bae1c34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:32.446957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2c934121-d5b1-4fa0-b5c2-7b804bae1c34 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:32.448623 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2c934121-d5b1-4fa0-b5c2-7b804bae1c34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:32.514906 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8127247-f299-45cd-a80a-4b28178742b9 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:32.516112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 75/149] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:32.516964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c934121-d5b1-4fa0-b5c2-7b804bae1c34 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:32.546193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:32.555544 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:32.570702 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fd1f3c-9631-404f-b505-ae4179c8b35f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:32.571174 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fd1f3c-9631-404f-b505-ae4179c8b35f None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:32.571460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fd1f3c-9631-404f-b505-ae4179c8b35f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:32.571736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fd1f3c-9631-404f-b505-ae4179c8b35f None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:32.572480 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82fd1f3c-9631-404f-b505-ae4179c8b35f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:32.574578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82fd1f3c-9631-404f-b505-ae4179c8b35f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:32.606944 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 75/150] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:43:32 2022] POST /identity/v3/users => generated 297 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:43:32.676567 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c934121-d5b1-4fa0-b5c2-7b804bae1c34 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:32.731088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6d12a8-6dcf-4a6f-ae0f-33ff7613a0cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:32.732185 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6d12a8-6dcf-4a6f-ae0f-33ff7613a0cc None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:32.732185 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6d12a8-6dcf-4a6f-ae0f-33ff7613a0cc None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:32.732185 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6d12a8-6dcf-4a6f-ae0f-33ff7613a0cc None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:32.732909 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6d12a8-6dcf-4a6f-ae0f-33ff7613a0cc None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:32.733222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6d12a8-6dcf-4a6f-ae0f-33ff7613a0cc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:32.734691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6d12a8-6dcf-4a6f-ae0f-33ff7613a0cc None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:32.744911 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 76/151] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:43:32 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:32.751399 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fd1f3c-9631-404f-b505-ae4179c8b35f None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:32.796511 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6d12a8-6dcf-4a6f-ae0f-33ff7613a0cc None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:32.881094 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fd1f3c-9631-404f-b505-ae4179c8b35f None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:32.916593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:32.932067 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6d12a8-6dcf-4a6f-ae0f-33ff7613a0cc None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:32.934121 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c571381-a9c6-4218-a596-ac1b86b24070 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:32.934782 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c571381-a9c6-4218-a596-ac1b86b24070 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:32.935105 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c571381-a9c6-4218-a596-ac1b86b24070 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:32.935379 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c571381-a9c6-4218-a596-ac1b86b24070 None admin] PATH_INFO: `/v3/users/1aa56909a1584148addcab430849c1e4` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:32.962226 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c571381-a9c6-4218-a596-ac1b86b24070 None admin] RBAC: Authorizing `identity:get_user(user_id=1aa56909a1584148addcab430849c1e4)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:32.964392 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c571381-a9c6-4218-a596-ac1b86b24070 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:32.965774 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 76/152] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:32 2022] GET /identity/v3/users/1aa56909a1584148addcab430849c1e4 => generated 283 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:32.971851 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c571381-a9c6-4218-a596-ac1b86b24070 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:32.986679 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96b7184-fac8-4b65-a25f-e28df3d8a4e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:32.987318 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96b7184-fac8-4b65-a25f-e28df3d8a4e3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:32.987568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96b7184-fac8-4b65-a25f-e28df3d8a4e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:32.987840 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96b7184-fac8-4b65-a25f-e28df3d8a4e3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:33.001785 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96b7184-fac8-4b65-a25f-e28df3d8a4e3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:33.006908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-d96b7184-fac8-4b65-a25f-e28df3d8a4e3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:43:33.006908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:33.006908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:33.006908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:33.006908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:33.006908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:33.006908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:33.006908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:33.006908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:33.006908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:33.006908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:33.006908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:33.006908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:33.006908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:33.006908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:33.006908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:33.008499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:33.008499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:33.008499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:33.008499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:33.008499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:33.008499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:33.008499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:33.008499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:33.008499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:33.008499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:33.008499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:33.008499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:33.008499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:33.008499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:33.008499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:33.008499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:33.009577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:33.009577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:33.009577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:33.009577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:33.009577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:33.009577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:33.009577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:33.009577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:33.009577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:33.009577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:33.009577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:43:33.009577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:33.009577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 77/153] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:43:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:43:33.014553 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96b7184-fac8-4b65-a25f-e28df3d8a4e3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:33.036251 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:33.141291 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96b7184-fac8-4b65-a25f-e28df3d8a4e3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:33.141877 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c571381-a9c6-4218-a596-ac1b86b24070 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:33.191062 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6c6138-431d-4ae1-8b08-a37b1109cab2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:33.192032 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cfa4c4-d0b6-4278-9bad-7e5cc5c37991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:33.192410 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6c6138-431d-4ae1-8b08-a37b1109cab2 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:33.192898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6c6138-431d-4ae1-8b08-a37b1109cab2 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:33.193073 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cfa4c4-d0b6-4278-9bad-7e5cc5c37991 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:33.193533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cfa4c4-d0b6-4278-9bad-7e5cc5c37991 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:33.193650 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6c6138-431d-4ae1-8b08-a37b1109cab2 None admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:33.194029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cfa4c4-d0b6-4278-9bad-7e5cc5c37991 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:33.195438 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cfa4c4-d0b6-4278-9bad-7e5cc5c37991 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:33.195683 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6c6138-431d-4ae1-8b08-a37b1109cab2 None admin] RBAC: Authorizing `identity:get_project(project_id=06615629856547fd9da00a208187c681)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:33.195987 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cfa4c4-d0b6-4278-9bad-7e5cc5c37991 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:33.197584 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6c6138-431d-4ae1-8b08-a37b1109cab2 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:33.199017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cfa4c4-d0b6-4278-9bad-7e5cc5c37991 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:33.200634 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 77/154] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:32 2022] GET /identity/v3/projects/06615629856547fd9da00a208187c681 => generated 302 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:33.211002 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6c6138-431d-4ae1-8b08-a37b1109cab2 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:33.222903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 78/155] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:33.228955 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cfa4c4-d0b6-4278-9bad-7e5cc5c37991 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:33.279380 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:33.335305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6c6138-431d-4ae1-8b08-a37b1109cab2 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:33.337129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:33.388697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd0d274-2876-4245-b921-951e35830bfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:33.389480 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd0d274-2876-4245-b921-951e35830bfd None admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:33.389765 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd0d274-2876-4245-b921-951e35830bfd None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:33.390334 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd0d274-2876-4245-b921-951e35830bfd None admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681/users/1aa56909a1584148addcab430849c1e4/roles/d9dcb4cfc0a245a98a7d96c100a497a1` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:33.392903 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd0d274-2876-4245-b921-951e35830bfd None admin] RBAC: Authorizing `identity:create_grant(project_id=06615629856547fd9da00a208187c681, user_id=1aa56909a1584148addcab430849c1e4, role_id=d9dcb4cfc0a245a98a7d96c100a497a1)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:33.394881 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd0d274-2876-4245-b921-951e35830bfd None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:33.420996 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cfa4c4-d0b6-4278-9bad-7e5cc5c37991 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:33.421222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 78/156] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:43:33 2022] PUT /identity/v3/projects/06615629856547fd9da00a208187c681/users/1aa56909a1584148addcab430849c1e4/roles/d9dcb4cfc0a245a98a7d96c100a497a1 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:43:33.467403 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee10d15-8bc8-45ae-bb66-3b9fbae8e401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:33.468349 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee10d15-8bc8-45ae-bb66-3b9fbae8e401 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:33.468776 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee10d15-8bc8-45ae-bb66-3b9fbae8e401 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:33.469339 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee10d15-8bc8-45ae-bb66-3b9fbae8e401 None admin] PATH_INFO: `/v3/users/1aa56909a1584148addcab430849c1e4` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:33.470880 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee10d15-8bc8-45ae-bb66-3b9fbae8e401 None admin] RBAC: Authorizing `identity:get_user(user_id=1aa56909a1584148addcab430849c1e4)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:33.472750 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee10d15-8bc8-45ae-bb66-3b9fbae8e401 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:33.474716 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 79/157] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:33 2022] GET /identity/v3/users/1aa56909a1584148addcab430849c1e4 => generated 283 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:33.483792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd0d274-2876-4245-b921-951e35830bfd None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:33.621298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd0d274-2876-4245-b921-951e35830bfd None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:33.671706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410ddc85-ae77-4579-916d-5ead17106e85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:33.672836 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410ddc85-ae77-4579-916d-5ead17106e85 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:33.673342 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410ddc85-ae77-4579-916d-5ead17106e85 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:33.673846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410ddc85-ae77-4579-916d-5ead17106e85 None admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:33.675862 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410ddc85-ae77-4579-916d-5ead17106e85 None admin] RBAC: Authorizing `identity:get_project(project_id=06615629856547fd9da00a208187c681)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:33.678736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410ddc85-ae77-4579-916d-5ead17106e85 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:33.682092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 79/158] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:33 2022] GET /identity/v3/projects/06615629856547fd9da00a208187c681 => generated 302 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:33.688256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee10d15-8bc8-45ae-bb66-3b9fbae8e401 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:33.810477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:33.849107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee10d15-8bc8-45ae-bb66-3b9fbae8e401 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:33.891463 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441b98f3-ff95-4ba4-920b-1ad0ac440855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:33.892057 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441b98f3-ff95-4ba4-920b-1ad0ac440855 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:33.892347 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441b98f3-ff95-4ba4-920b-1ad0ac440855 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:33.892628 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441b98f3-ff95-4ba4-920b-1ad0ac440855 None admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681/users/1aa56909a1584148addcab430849c1e4/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:33.894705 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441b98f3-ff95-4ba4-920b-1ad0ac440855 None admin] RBAC: Authorizing `identity:create_grant(project_id=06615629856547fd9da00a208187c681, user_id=1aa56909a1584148addcab430849c1e4, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:33.896212 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441b98f3-ff95-4ba4-920b-1ad0ac440855 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:33.911869 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 80/159] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:43:33 2022] PUT /identity/v3/projects/06615629856547fd9da00a208187c681/users/1aa56909a1584148addcab430849c1e4/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:43:35.177705 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3030566d-1acb-4580-8942-659d2c999844 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:35.178222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3030566d-1acb-4580-8942-659d2c999844 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:35.178299 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3030566d-1acb-4580-8942-659d2c999844 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:35.179644 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 80/160] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:43:35.185349 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e7783c-7fc8-49ed-adb8-bf4892a38f86 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:35.185813 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e7783c-7fc8-49ed-adb8-bf4892a38f86 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:35.186133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e7783c-7fc8-49ed-adb8-bf4892a38f86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:35.206556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f0e7783c-7fc8-49ed-adb8-bf4892a38f86 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:35.207794 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f0e7783c-7fc8-49ed-adb8-bf4892a38f86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:35.316374 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 81/161] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:35.555120 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3610cb18-3923-4e5c-9dc3-6c3ab4865d48 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:35.555404 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3610cb18-3923-4e5c-9dc3-6c3ab4865d48 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:35.555919 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3610cb18-3923-4e5c-9dc3-6c3ab4865d48 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:35.558025 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 81/162] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:43:35.566903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb092c5-e5f9-4508-a4a4-4d802d8644c5 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:35.567747 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb092c5-e5f9-4508-a4a4-4d802d8644c5 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:35.568521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb092c5-e5f9-4508-a4a4-4d802d8644c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:35.595305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3610cb18-3923-4e5c-9dc3-6c3ab4865d48 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:35.600568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-abb092c5-e5f9-4508-a4a4-4d802d8644c5 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:35.602549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-abb092c5-e5f9-4508-a4a4-4d802d8644c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:35.632596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:35.723541 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3610cb18-3923-4e5c-9dc3-6c3ab4865d48 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:35.761220 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:35.770751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1eaba6b-bf72-42e7-8846-696680852449 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:35.771824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eaba6b-bf72-42e7-8846-696680852449 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:35.772338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eaba6b-bf72-42e7-8846-696680852449 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:35.772882 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eaba6b-bf72-42e7-8846-696680852449 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:35.774269 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1eaba6b-bf72-42e7-8846-696680852449 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:35.775059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 82/163] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:35.776607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1eaba6b-bf72-42e7-8846-696680852449 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:35.778468 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 82/164] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:43:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:35.785301 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb092c5-e5f9-4508-a4a4-4d802d8644c5 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:35.911753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb092c5-e5f9-4508-a4a4-4d802d8644c5 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:35.952566 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057c96a2-35c9-41f7-a3b7-c7a0767bbe8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:35.953663 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057c96a2-35c9-41f7-a3b7-c7a0767bbe8f None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:35.954161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057c96a2-35c9-41f7-a3b7-c7a0767bbe8f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:35.954718 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057c96a2-35c9-41f7-a3b7-c7a0767bbe8f None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:35.955879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057c96a2-35c9-41f7-a3b7-c7a0767bbe8f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:35.957556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057c96a2-35c9-41f7-a3b7-c7a0767bbe8f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:35.983551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 83/165] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:43:35 2022] POST /identity/v3/users => generated 297 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:43:36.205337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a58e86-9042-4923-832c-f19f8ff3f55c None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:36.205831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a58e86-9042-4923-832c-f19f8ff3f55c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:36.205831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a58e86-9042-4923-832c-f19f8ff3f55c None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:36.207451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 83/166] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:36.213269 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7252629e-8bf5-4f18-b406-77cc30e33c6a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:36.213863 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7252629e-8bf5-4f18-b406-77cc30e33c6a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:36.213948 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7252629e-8bf5-4f18-b406-77cc30e33c6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:36.235368 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-7252629e-8bf5-4f18-b406-77cc30e33c6a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:36.236546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7252629e-8bf5-4f18-b406-77cc30e33c6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:36.291309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578fb7a6-6cad-43d8-b9b6-c1668fd662c6 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:36.291693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578fb7a6-6cad-43d8-b9b6-c1668fd662c6 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:36.291942 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578fb7a6-6cad-43d8-b9b6-c1668fd662c6 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:36.293298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 84/167] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:43:36.299422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a177a33-b4a9-4448-892d-6d9448365775 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:36.299743 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a177a33-b4a9-4448-892d-6d9448365775 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:36.299977 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a177a33-b4a9-4448-892d-6d9448365775 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:36.300289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:36.323552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-6a177a33-b4a9-4448-892d-6d9448365775 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:36.324935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6a177a33-b4a9-4448-892d-6d9448365775 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:36.330358 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 84/168] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:36.331249 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7252629e-8bf5-4f18-b406-77cc30e33c6a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:36.431550 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 85/169] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:43:36.508008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7252629e-8bf5-4f18-b406-77cc30e33c6a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:36.575554 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a712465-4356-41e6-8bad-8abd3d453dfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:36.576638 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a712465-4356-41e6-8bad-8abd3d453dfe None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:36.577114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a712465-4356-41e6-8bad-8abd3d453dfe None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:36.577563 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a712465-4356-41e6-8bad-8abd3d453dfe None admin] PATH_INFO: `/v3/roles/member` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:36.597580 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a712465-4356-41e6-8bad-8abd3d453dfe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:36.607930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-6a712465-4356-41e6-8bad-8abd3d453dfe None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:36.607930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:36.607930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:36.607930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:36.607930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:36.607930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:36.607930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:36.607930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:36.607930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:36.607930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:36.607930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:36.607930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:36.607930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:36.607930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:36.607930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:36.607930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:36.609173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:36.609173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:36.609173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:36.609173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:36.609173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:36.609173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:36.609173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:36.609173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:36.609173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:36.609173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:36.609173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:36.609173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:36.609173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:36.609173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:36.609173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:36.609173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:36.609997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:36.609997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:36.609997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:36.609997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:36.609997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:36.609997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:36.609997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:36.609997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:36.609997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:36.609997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:36.609997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:36.609997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:36.609997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 85/170] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:36 2022] GET /identity/v3/roles/member => generated 84 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:36.620485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a177a33-b4a9-4448-892d-6d9448365775 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:36.668449 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a712465-4356-41e6-8bad-8abd3d453dfe None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:36.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:36.780587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:36.846181 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a177a33-b4a9-4448-892d-6d9448365775 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:36.872165 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a712465-4356-41e6-8bad-8abd3d453dfe None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:36.896186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d66213-63e8-4c4d-abf1-aea4f855723a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:36.896834 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d66213-63e8-4c4d-abf1-aea4f855723a None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:36.897109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d66213-63e8-4c4d-abf1-aea4f855723a None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:36.897379 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d66213-63e8-4c4d-abf1-aea4f855723a None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:36.898064 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d66213-63e8-4c4d-abf1-aea4f855723a None admin] RBAC: Adding query filter params (name=member) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:36.898365 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d66213-63e8-4c4d-abf1-aea4f855723a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:36.899815 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d66213-63e8-4c4d-abf1-aea4f855723a None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:36.912520 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 86/171] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:43:36 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:36.913693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d66213-63e8-4c4d-abf1-aea4f855723a None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:36.921310 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe66c57e-b2be-40d1-8f51-eb8afe539a16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:36.922606 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe66c57e-b2be-40d1-8f51-eb8afe539a16 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:36.923168 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe66c57e-b2be-40d1-8f51-eb8afe539a16 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:36.923787 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe66c57e-b2be-40d1-8f51-eb8afe539a16 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:36.939250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe66c57e-b2be-40d1-8f51-eb8afe539a16 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:36.946178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-fe66c57e-b2be-40d1-8f51-eb8afe539a16 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:43:36.946178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:36.946178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:36.946178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:36.946178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:36.946178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:36.946178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:36.946178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:36.946178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:36.946178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:36.946178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:36.946178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:36.946178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:36.946178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:36.946178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:36.946178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:36.947306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:36.947306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:36.947306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:36.947306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:36.947306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:36.947306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:36.947306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:36.947306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:36.947306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:36.947306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:36.947306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:36.947306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:36.947306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:36.947306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:36.947306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:36.947306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:36.948956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:36.948956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:36.948956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:36.948956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:36.948956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:36.948956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:36.948956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:36.948956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:36.948956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:36.948956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:36.948956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:43:36.948956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:36.951643 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 86/172] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:43:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:36.952726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe66c57e-b2be-40d1-8f51-eb8afe539a16 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:37.039748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:37.056970 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d66213-63e8-4c4d-abf1-aea4f855723a None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:37.077823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe66c57e-b2be-40d1-8f51-eb8afe539a16 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:37.081915 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:37.109728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eac8a82-8e02-41f3-8552-f86fae884a42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:37.111133 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eac8a82-8e02-41f3-8552-f86fae884a42 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:37.111790 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eac8a82-8e02-41f3-8552-f86fae884a42 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:37.112471 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eac8a82-8e02-41f3-8552-f86fae884a42 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:37.125660 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c21ed8-c38c-4bd6-956f-cace5cce9eeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:37.126319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c21ed8-c38c-4bd6-956f-cace5cce9eeb None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:37.126595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c21ed8-c38c-4bd6-956f-cace5cce9eeb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:37.126891 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c21ed8-c38c-4bd6-956f-cace5cce9eeb None admin] PATH_INFO: `/v3/users/74ebd9bcd208442bbf14f5d46a426f42` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:37.129438 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eac8a82-8e02-41f3-8552-f86fae884a42 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:37.136222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-6eac8a82-8e02-41f3-8552-f86fae884a42 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:43:37.136222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:37.136222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:37.136222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:37.136222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:37.136222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:37.136222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:37.136222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:37.136222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:37.136222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:37.136222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:37.136222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:37.136222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:37.136222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:37.136222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:37.136222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:37.137270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:37.137270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:37.137270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:37.137270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:37.137270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:37.137270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:37.137270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:37.137270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:37.137270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:37.137270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:37.137270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:37.137270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:37.137270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:37.137270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:37.137270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:37.137270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:37.138071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:37.138071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:37.138071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:37.138071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:37.138071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:37.138071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:37.138071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:37.138071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:37.138071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:37.138071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:37.138071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:43:37.138071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:37.140664 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 87/173] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 16 00:43:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:37.141805 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eac8a82-8e02-41f3-8552-f86fae884a42 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:37.142841 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c21ed8-c38c-4bd6-956f-cace5cce9eeb None admin] RBAC: Authorizing `identity:get_user(user_id=74ebd9bcd208442bbf14f5d46a426f42)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:37.144373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c21ed8-c38c-4bd6-956f-cace5cce9eeb None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:37.147980 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 87/174] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:36 2022] GET /identity/v3/users/74ebd9bcd208442bbf14f5d46a426f42 => generated 297 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:37.148874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c21ed8-c38c-4bd6-956f-cace5cce9eeb None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:37.282317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eac8a82-8e02-41f3-8552-f86fae884a42 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:37.288332 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c21ed8-c38c-4bd6-956f-cace5cce9eeb None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:37.330515 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c1bf9f-b5c6-4b24-a6f0-2c136f70cd08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:37.331256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c1bf9f-b5c6-4b24-a6f0-2c136f70cd08 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:37.331701 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c1bf9f-b5c6-4b24-a6f0-2c136f70cd08 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:37.331987 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c1bf9f-b5c6-4b24-a6f0-2c136f70cd08 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:37.332837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c1bf9f-b5c6-4b24-a6f0-2c136f70cd08 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:37.333369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c1bf9f-b5c6-4b24-a6f0-2c136f70cd08 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:37.334746 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c1bf9f-b5c6-4b24-a6f0-2c136f70cd08 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:37.339828 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ffbff3-64c5-4f7b-bfb7-426f3730c33e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:37.341215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ffbff3-64c5-4f7b-bfb7-426f3730c33e None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:37.341739 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ffbff3-64c5-4f7b-bfb7-426f3730c33e None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:37.342458 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ffbff3-64c5-4f7b-bfb7-426f3730c33e None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:37.343535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ffbff3-64c5-4f7b-bfb7-426f3730c33e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:37.344198 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ffbff3-64c5-4f7b-bfb7-426f3730c33e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:37.348105 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ffbff3-64c5-4f7b-bfb7-426f3730c33e None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:37.352460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 88/175] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:43:37 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:37.353930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c1bf9f-b5c6-4b24-a6f0-2c136f70cd08 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:37.360168 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 88/176] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:37.365979 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ffbff3-64c5-4f7b-bfb7-426f3730c33e None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:37.404695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:37.409741 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:37.528152 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c1bf9f-b5c6-4b24-a6f0-2c136f70cd08 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:37.533940 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ffbff3-64c5-4f7b-bfb7-426f3730c33e None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:37.578759 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c68a8aa-4f95-4563-9ee6-aad074c13afa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:37.579579 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c68a8aa-4f95-4563-9ee6-aad074c13afa None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:37.580218 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c68a8aa-4f95-4563-9ee6-aad074c13afa None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:37.580684 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c68a8aa-4f95-4563-9ee6-aad074c13afa None admin] PATH_INFO: `/v3/users/1aa56909a1584148addcab430849c1e4` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:37.581886 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c68a8aa-4f95-4563-9ee6-aad074c13afa None admin] RBAC: Authorizing `identity:get_user(user_id=1aa56909a1584148addcab430849c1e4)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:37.583497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c68a8aa-4f95-4563-9ee6-aad074c13afa None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:37.584532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d437ff25-4ce2-4da0-bd8e-659b1e783153 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:37.585580 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 89/177] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:37 2022] GET /identity/v3/users/1aa56909a1584148addcab430849c1e4 => generated 283 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:37.586220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c68a8aa-4f95-4563-9ee6-aad074c13afa None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:37.586674 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d437ff25-4ce2-4da0-bd8e-659b1e783153 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:37.587271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d437ff25-4ce2-4da0-bd8e-659b1e783153 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:37.587742 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d437ff25-4ce2-4da0-bd8e-659b1e783153 None admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:37.589552 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d437ff25-4ce2-4da0-bd8e-659b1e783153 None admin] RBAC: Authorizing `identity:get_project(project_id=06615629856547fd9da00a208187c681)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:37.591464 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d437ff25-4ce2-4da0-bd8e-659b1e783153 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:37.594966 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 89/178] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:37 2022] GET /identity/v3/projects/06615629856547fd9da00a208187c681 => generated 302 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:37.596783 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d437ff25-4ce2-4da0-bd8e-659b1e783153 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:37.747287 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:37.751560 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:37.752583 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c68a8aa-4f95-4563-9ee6-aad074c13afa None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:37.772828 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d437ff25-4ce2-4da0-bd8e-659b1e783153 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:37.826201 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a42252d-f6ba-49f9-a3ca-2d9c9a137589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:37.828054 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a42252d-f6ba-49f9-a3ca-2d9c9a137589 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:37.828926 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a42252d-f6ba-49f9-a3ca-2d9c9a137589 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:37.830061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a42252d-f6ba-49f9-a3ca-2d9c9a137589 None admin] PATH_INFO: `/v3/users/1aa56909a1584148addcab430849c1e4` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:37.833126 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a42252d-f6ba-49f9-a3ca-2d9c9a137589 None admin] RBAC: Authorizing `identity:get_user(user_id=1aa56909a1584148addcab430849c1e4)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:37.838414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a42252d-f6ba-49f9-a3ca-2d9c9a137589 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:37.841465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 90/179] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:37 2022] GET /identity/v3/users/1aa56909a1584148addcab430849c1e4 => generated 283 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:37.842743 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a42252d-f6ba-49f9-a3ca-2d9c9a137589 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:37.851457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5027e4b-7964-45d4-aa85-db6124efbe53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:37.852453 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5027e4b-7964-45d4-aa85-db6124efbe53 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:37.852802 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5027e4b-7964-45d4-aa85-db6124efbe53 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:37.853175 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5027e4b-7964-45d4-aa85-db6124efbe53 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:37.964848 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5027e4b-7964-45d4-aa85-db6124efbe53 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:37.967873 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5027e4b-7964-45d4-aa85-db6124efbe53 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:38.016680 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a42252d-f6ba-49f9-a3ca-2d9c9a137589 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:38.063179 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d47d999-9431-4543-9ed4-043db113bd72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:38.064434 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d47d999-9431-4543-9ed4-043db113bd72 None admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:38.064924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d47d999-9431-4543-9ed4-043db113bd72 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:38.065451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d47d999-9431-4543-9ed4-043db113bd72 None admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681/users/74ebd9bcd208442bbf14f5d46a426f42/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:38.067812 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d47d999-9431-4543-9ed4-043db113bd72 None admin] RBAC: Authorizing `identity:create_grant(project_id=06615629856547fd9da00a208187c681, user_id=74ebd9bcd208442bbf14f5d46a426f42, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:38.069663 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d47d999-9431-4543-9ed4-043db113bd72 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:38.090058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 91/180] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:43:37 2022] PUT /identity/v3/projects/06615629856547fd9da00a208187c681/users/74ebd9bcd208442bbf14f5d46a426f42/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:43:38.091376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d47d999-9431-4543-9ed4-043db113bd72 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:38.110131 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:38.188648 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:38.211536 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 90/181] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:43:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 615 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:43:38.217042 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5027e4b-7964-45d4-aa85-db6124efbe53 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:38.332831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d47d999-9431-4543-9ed4-043db113bd72 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:38.415435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10334f83-5d65-41a1-bbc8-f61f634da075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:38.417734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10334f83-5d65-41a1-bbc8-f61f634da075 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:38.418227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10334f83-5d65-41a1-bbc8-f61f634da075 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:38.418526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10334f83-5d65-41a1-bbc8-f61f634da075 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:38.461015 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5027e4b-7964-45d4-aa85-db6124efbe53 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:38.523556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8e2d00-278d-4fee-b357-18a2ba373ea3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:38.524530 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8e2d00-278d-4fee-b357-18a2ba373ea3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:38.524992 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8e2d00-278d-4fee-b357-18a2ba373ea3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:38.526181 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8e2d00-278d-4fee-b357-18a2ba373ea3 None admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:38.528080 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8e2d00-278d-4fee-b357-18a2ba373ea3 None admin] RBAC: Authorizing `identity:get_project(project_id=06615629856547fd9da00a208187c681)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:38.529867 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8e2d00-278d-4fee-b357-18a2ba373ea3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:38.532549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 91/182] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:38 2022] GET /identity/v3/projects/06615629856547fd9da00a208187c681 => generated 302 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:38.543360 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8e2d00-278d-4fee-b357-18a2ba373ea3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:38.574654 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10334f83-5d65-41a1-bbc8-f61f634da075 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:38.576675 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10334f83-5d65-41a1-bbc8-f61f634da075 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:38.615064 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:38.666203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:38.785409 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8e2d00-278d-4fee-b357-18a2ba373ea3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:38.794925 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 92/183] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:43:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 698 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:43:38.809533 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10334f83-5d65-41a1-bbc8-f61f634da075 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:38.876653 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21a6608-3fd1-480b-b57c-7faa75aca778 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:38.877289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21a6608-3fd1-480b-b57c-7faa75aca778 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:38.877570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21a6608-3fd1-480b-b57c-7faa75aca778 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:38.877844 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21a6608-3fd1-480b-b57c-7faa75aca778 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:38.879011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21a6608-3fd1-480b-b57c-7faa75aca778 None admin] RBAC: Adding query filter params (role.id=d9dcb4cfc0a245a98a7d96c100a497a1, scope.project.id=06615629856547fd9da00a208187c681, user.id=1aa56909a1584148addcab430849c1e4) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:38.879011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21a6608-3fd1-480b-b57c-7faa75aca778 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:38.880584 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21a6608-3fd1-480b-b57c-7faa75aca778 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:38.897693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 92/184] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:43:38 2022] GET /identity/v3/role_assignments?user.id=1aa56909a1584148addcab430849c1e4&scope.project.id=06615629856547fd9da00a208187c681&role.id=d9dcb4cfc0a245a98a7d96c100a497a1 => generated 608 bytes in 355 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:38.899054 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ceb1d2-217f-4f7a-86c7-e33f3ccc87f0 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:38.899346 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ceb1d2-217f-4f7a-86c7-e33f3ccc87f0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:38.906897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ceb1d2-217f-4f7a-86c7-e33f3ccc87f0 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:38.906897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 93/185] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:38.916504 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7083be-9a2b-4096-a165-f0008bdfd156 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:38.916790 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7083be-9a2b-4096-a165-f0008bdfd156 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:38.917764 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7083be-9a2b-4096-a165-f0008bdfd156 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:38.950904 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-1e7083be-9a2b-4096-a165-f0008bdfd156 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:38.953381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1e7083be-9a2b-4096-a165-f0008bdfd156 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:39.062048 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10334f83-5d65-41a1-bbc8-f61f634da075 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:39.074105 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:39.123393 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 94/186] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:39.124479 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a46fff-a676-482e-9acc-8fb65834ff41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:39.125633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a46fff-a676-482e-9acc-8fb65834ff41 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:39.126123 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a46fff-a676-482e-9acc-8fb65834ff41 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:39.126605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a46fff-a676-482e-9acc-8fb65834ff41 None admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:39.128824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a46fff-a676-482e-9acc-8fb65834ff41 None admin] RBAC: Authorizing `identity:get_project(project_id=06615629856547fd9da00a208187c681)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:39.131668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a46fff-a676-482e-9acc-8fb65834ff41 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:39.139328 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 93/187] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:38 2022] GET /identity/v3/projects/06615629856547fd9da00a208187c681 => generated 302 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:39.145981 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7083be-9a2b-4096-a165-f0008bdfd156 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:39.281349 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:39.368858 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7083be-9a2b-4096-a165-f0008bdfd156 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:39.391517 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f98a16-5789-432e-9594-049f69f93093 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:39.392264 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f98a16-5789-432e-9594-049f69f93093 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:39.392264 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f98a16-5789-432e-9594-049f69f93093 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:39.394641 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 94/188] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:43:39.402515 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749e6385-65e4-4108-83a6-eb4df1190b93 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:39.402515 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749e6385-65e4-4108-83a6-eb4df1190b93 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:39.402515 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749e6385-65e4-4108-83a6-eb4df1190b93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:39.436951 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-749e6385-65e4-4108-83a6-eb4df1190b93 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:39.440769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-749e6385-65e4-4108-83a6-eb4df1190b93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:39.448128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2708da-8086-46a4-b165-ac9438e4a7a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:39.449143 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2708da-8086-46a4-b165-ac9438e4a7a9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:39.449628 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2708da-8086-46a4-b165-ac9438e4a7a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:39.450101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2708da-8086-46a4-b165-ac9438e4a7a9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:39.452420 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2708da-8086-46a4-b165-ac9438e4a7a9 None admin] RBAC: Adding query filter params (role.id=c5cd777d70a649028f970fd986dc484f, scope.project.id=06615629856547fd9da00a208187c681, user.id=1aa56909a1584148addcab430849c1e4) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:39.452927 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2708da-8086-46a4-b165-ac9438e4a7a9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:39.457397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2708da-8086-46a4-b165-ac9438e4a7a9 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:39.472810 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 95/189] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:43:39 2022] GET /identity/v3/role_assignments?user.id=1aa56909a1584148addcab430849c1e4&scope.project.id=06615629856547fd9da00a208187c681&role.id=c5cd777d70a649028f970fd986dc484f => generated 608 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:39.537010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 95/190] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:43:39.735325 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2708da-8086-46a4-b165-ac9438e4a7a9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:39.827619 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749e6385-65e4-4108-83a6-eb4df1190b93 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:39.936894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2708da-8086-46a4-b165-ac9438e4a7a9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:39.945505 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749e6385-65e4-4108-83a6-eb4df1190b93 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:39.954775 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:39.969327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:39.986173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307d8252-5758-4341-ad77-dfe1bd646bc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:39.986825 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307d8252-5758-4341-ad77-dfe1bd646bc9 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:39.987140 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307d8252-5758-4341-ad77-dfe1bd646bc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:39.987805 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307d8252-5758-4341-ad77-dfe1bd646bc9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:39.988547 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307d8252-5758-4341-ad77-dfe1bd646bc9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:39.990023 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307d8252-5758-4341-ad77-dfe1bd646bc9 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:39.991641 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 96/191] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:43:39 2022] GET /identity/v3/domains/default => generated 199 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:39.993400 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60881750-e916-46b1-b009-788c7deea9ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:39.994048 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60881750-e916-46b1-b009-788c7deea9ad None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:39.994316 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60881750-e916-46b1-b009-788c7deea9ad None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:39.994589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60881750-e916-46b1-b009-788c7deea9ad None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:39.997309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307d8252-5758-4341-ad77-dfe1bd646bc9 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:40.009995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60881750-e916-46b1-b009-788c7deea9ad None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:40.018158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-60881750-e916-46b1-b009-788c7deea9ad None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 16 00:43:40.018158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:40.018158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:40.018158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:40.018158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:40.018158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:40.018158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:40.018158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:40.018158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:40.018158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:40.018158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:40.018158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:40.018158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:40.018158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:40.018158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:40.018158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:40.020260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:40.020260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:40.020260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:40.020260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:40.020260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:40.020260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:40.020260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:40.020260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:40.020260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:40.020260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:40.020260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:40.020260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:40.020260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:40.020260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:40.020260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:40.020260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:40.021865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:40.021865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:40.021865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:40.021865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:40.021865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:40.021865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:40.021865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:40.021865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:40.021865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:40.021865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:40.021865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 16 00:43:40.021865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:40.021865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 96/192] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:39 2022] GET /identity/v3/roles/reader => generated 84 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:40.025562 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60881750-e916-46b1-b009-788c7deea9ad None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:40.141634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60881750-e916-46b1-b009-788c7deea9ad None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:40.149224 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307d8252-5758-4341-ad77-dfe1bd646bc9 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:40.179180 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ba2f56-6699-4952-b615-dc469cb67759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:40.179822 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ba2f56-6699-4952-b615-dc469cb67759 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:40.180104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ba2f56-6699-4952-b615-dc469cb67759 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:40.180378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ba2f56-6699-4952-b615-dc469cb67759 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:40.181093 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ba2f56-6699-4952-b615-dc469cb67759 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:40.181397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ba2f56-6699-4952-b615-dc469cb67759 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:40.183444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ba2f56-6699-4952-b615-dc469cb67759 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:40.194274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 97/193] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:43:40 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:40.200000 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ba2f56-6699-4952-b615-dc469cb67759 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:40.215471 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08762773-12fa-4e6c-bd21-1f6855a66f3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:40.216114 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08762773-12fa-4e6c-bd21-1f6855a66f3d None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:40.216576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08762773-12fa-4e6c-bd21-1f6855a66f3d None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:40.216866 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08762773-12fa-4e6c-bd21-1f6855a66f3d None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:40.218015 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08762773-12fa-4e6c-bd21-1f6855a66f3d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:40.219867 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08762773-12fa-4e6c-bd21-1f6855a66f3d None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:40.246373 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 97/194] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:43:39 2022] POST /identity/v3/users => generated 293 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:43:40.277919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:40.393277 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ba2f56-6699-4952-b615-dc469cb67759 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:40.462367 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64582997-2466-4e2d-bb91-1a137d431bb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:40.463153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64582997-2466-4e2d-bb91-1a137d431bb8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:40.463464 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64582997-2466-4e2d-bb91-1a137d431bb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:40.463870 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64582997-2466-4e2d-bb91-1a137d431bb8 None admin] PATH_INFO: `/v3/users/7beaa53b40cd4f2e925660bc4bfb61a7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:40.480390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64582997-2466-4e2d-bb91-1a137d431bb8 None admin] RBAC: Authorizing `identity:get_user(user_id=7beaa53b40cd4f2e925660bc4bfb61a7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:40.481974 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64582997-2466-4e2d-bb91-1a137d431bb8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:40.483799 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 98/195] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:40 2022] GET /identity/v3/users/7beaa53b40cd4f2e925660bc4bfb61a7 => generated 297 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:40.489563 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08762773-12fa-4e6c-bd21-1f6855a66f3d None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:40.527669 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:40.652464 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08762773-12fa-4e6c-bd21-1f6855a66f3d None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:40.715859 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04117f5e-b042-4d0d-86fe-b9592ce90976 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:40.717024 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04117f5e-b042-4d0d-86fe-b9592ce90976 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:40.717730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04117f5e-b042-4d0d-86fe-b9592ce90976 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:40.718217 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04117f5e-b042-4d0d-86fe-b9592ce90976 None admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:40.720293 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04117f5e-b042-4d0d-86fe-b9592ce90976 None admin] RBAC: Authorizing `identity:get_project(project_id=06615629856547fd9da00a208187c681)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:40.723144 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04117f5e-b042-4d0d-86fe-b9592ce90976 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:40.726475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 98/196] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:40 2022] GET /identity/v3/projects/06615629856547fd9da00a208187c681 => generated 302 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:40.736451 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64582997-2466-4e2d-bb91-1a137d431bb8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:40.926958 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:40.934695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64582997-2466-4e2d-bb91-1a137d431bb8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:41.000601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b450f294-b2b0-43f1-af43-92fd49d1dc32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:41.001723 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b450f294-b2b0-43f1-af43-92fd49d1dc32 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:41.002205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b450f294-b2b0-43f1-af43-92fd49d1dc32 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:41.002694 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b450f294-b2b0-43f1-af43-92fd49d1dc32 None admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681/users/7beaa53b40cd4f2e925660bc4bfb61a7/roles/1ac2f92c6eb34313ac9d9dfd4476663c` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:41.006430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b450f294-b2b0-43f1-af43-92fd49d1dc32 None admin] RBAC: Authorizing `identity:create_grant(project_id=06615629856547fd9da00a208187c681, user_id=7beaa53b40cd4f2e925660bc4bfb61a7, role_id=1ac2f92c6eb34313ac9d9dfd4476663c)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:41.009212 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b450f294-b2b0-43f1-af43-92fd49d1dc32 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:41.032932 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 99/197] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:43:40 2022] PUT /identity/v3/projects/06615629856547fd9da00a208187c681/users/7beaa53b40cd4f2e925660bc4bfb61a7/roles/1ac2f92c6eb34313ac9d9dfd4476663c => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:43:41.716003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5492861-1cd8-472b-a338-0367128ac6d0 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:41.716498 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5492861-1cd8-472b-a338-0367128ac6d0 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:41.716498 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5492861-1cd8-472b-a338-0367128ac6d0 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:41.718102 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 99/198] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:41.723781 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d182212-147f-499a-b345-1378dc4546f0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:41.724258 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d182212-147f-499a-b345-1378dc4546f0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:41.724362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d182212-147f-499a-b345-1378dc4546f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:41.744064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4d182212-147f-499a-b345-1378dc4546f0 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:41.745309 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4d182212-147f-499a-b345-1378dc4546f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:41.835555 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 100/199] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:42.108439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5492861-1cd8-472b-a338-0367128ac6d0 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:42.282628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:42.291542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5492861-1cd8-472b-a338-0367128ac6d0 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:42.355437 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091d91cc-ffa7-4623-af6d-02f3fe19aca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:42.356649 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091d91cc-ffa7-4623-af6d-02f3fe19aca1 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:42.357032 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091d91cc-ffa7-4623-af6d-02f3fe19aca1 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:42.357421 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091d91cc-ffa7-4623-af6d-02f3fe19aca1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:42.378785 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091d91cc-ffa7-4623-af6d-02f3fe19aca1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:42.387957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-091d91cc-ffa7-4623-af6d-02f3fe19aca1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:42.387957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:42.387957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:42.387957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:42.387957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:42.387957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:42.387957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:42.387957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:42.387957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:42.387957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:42.387957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:42.387957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:42.387957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:42.387957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:42.387957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:42.387957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:42.388991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:42.388991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:42.388991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:42.388991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:42.388991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:42.388991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:42.388991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:42.388991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:42.388991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:42.388991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:42.388991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:42.388991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:42.388991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:42.388991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:42.388991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:42.388991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:42.389797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:42.389797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:42.389797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:42.389797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:42.389797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:42.389797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:42.389797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:42.389797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:42.389797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:42.389797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:42.389797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:42.389797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:42.389797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 100/200] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:42 2022] GET /identity/v3/roles/member => generated 84 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:42.397741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d182212-147f-499a-b345-1378dc4546f0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:42.561272 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:42.585385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d182212-147f-499a-b345-1378dc4546f0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:42.649017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58b00a2-ea66-4c5e-a997-75bab16c8c98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:42.649982 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58b00a2-ea66-4c5e-a997-75bab16c8c98 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:42.650463 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58b00a2-ea66-4c5e-a997-75bab16c8c98 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:42.651497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58b00a2-ea66-4c5e-a997-75bab16c8c98 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:42.652753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58b00a2-ea66-4c5e-a997-75bab16c8c98 None admin] RBAC: Adding query filter params (name=member) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:42.653293 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58b00a2-ea66-4c5e-a997-75bab16c8c98 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:42.655847 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58b00a2-ea66-4c5e-a997-75bab16c8c98 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:42.672161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 101/201] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:43:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:42.681489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091d91cc-ffa7-4623-af6d-02f3fe19aca1 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:42.806362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f18cad-5eb3-453f-a634-9f295227d3f9 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:42.808302 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f18cad-5eb3-453f-a634-9f295227d3f9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:42.809011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f18cad-5eb3-453f-a634-9f295227d3f9 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:42.811341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 102/202] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:42 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:42.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cf8d18-37a7-4f08-844f-d82df9dfcf12 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:42.820687 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cf8d18-37a7-4f08-844f-d82df9dfcf12 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:42.821446 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cf8d18-37a7-4f08-844f-d82df9dfcf12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:42.855155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-91cf8d18-37a7-4f08-844f-d82df9dfcf12 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:42.856745 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091d91cc-ffa7-4623-af6d-02f3fe19aca1 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:42.857429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-91cf8d18-37a7-4f08-844f-d82df9dfcf12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:42.898800 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6607cd2-b5a3-4dcd-b11d-e5685aad2866 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:42.899593 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6607cd2-b5a3-4dcd-b11d-e5685aad2866 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:42.899890 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6607cd2-b5a3-4dcd-b11d-e5685aad2866 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:42.900061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6607cd2-b5a3-4dcd-b11d-e5685aad2866 None admin] PATH_INFO: `/v3/users/74ebd9bcd208442bbf14f5d46a426f42` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:42.901399 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6607cd2-b5a3-4dcd-b11d-e5685aad2866 None admin] RBAC: Authorizing `identity:get_user(user_id=74ebd9bcd208442bbf14f5d46a426f42)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:42.903113 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6607cd2-b5a3-4dcd-b11d-e5685aad2866 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:42.910017 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 101/203] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:42 2022] GET /identity/v3/users/74ebd9bcd208442bbf14f5d46a426f42 => generated 297 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:42.910017 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561022d5-3b08-4ea7-80b2-4c1ef72a82de None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:42.910017 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561022d5-3b08-4ea7-80b2-4c1ef72a82de None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:42.910017 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561022d5-3b08-4ea7-80b2-4c1ef72a82de None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:42.911022 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 102/204] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:42 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:42.912131 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561022d5-3b08-4ea7-80b2-4c1ef72a82de None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:42.961287 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:42.993427 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 103/205] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:43:43.000806 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838c04d5-8fa0-4d1f-bc69-291a1ec5a163 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:43.001428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838c04d5-8fa0-4d1f-bc69-291a1ec5a163 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:43.001921 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838c04d5-8fa0-4d1f-bc69-291a1ec5a163 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:43.028073 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-838c04d5-8fa0-4d1f-bc69-291a1ec5a163 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:43.030484 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-838c04d5-8fa0-4d1f-bc69-291a1ec5a163 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:43.103409 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561022d5-3b08-4ea7-80b2-4c1ef72a82de None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:43.130685 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 104/206] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:43.160983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab4b7dc-54af-4bdf-89a2-cd21152e0b96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:43.161981 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab4b7dc-54af-4bdf-89a2-cd21152e0b96 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:43.162494 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab4b7dc-54af-4bdf-89a2-cd21152e0b96 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:43.163063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab4b7dc-54af-4bdf-89a2-cd21152e0b96 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:43.245186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab4b7dc-54af-4bdf-89a2-cd21152e0b96 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:43.247130 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab4b7dc-54af-4bdf-89a2-cd21152e0b96 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:43.278893 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:43.365796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 103/207] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:43:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 454 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:43:43.372750 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838c04d5-8fa0-4d1f-bc69-291a1ec5a163 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:43.386796 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:43.512219 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838c04d5-8fa0-4d1f-bc69-291a1ec5a163 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:43.523874 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab4b7dc-54af-4bdf-89a2-cd21152e0b96 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:43.584053 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea55b4d1-932a-42de-ad6c-4738ca9bef00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:43.585148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea55b4d1-932a-42de-ad6c-4738ca9bef00 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:43.585643 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea55b4d1-932a-42de-ad6c-4738ca9bef00 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:43.586128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea55b4d1-932a-42de-ad6c-4738ca9bef00 None admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:43.588158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea55b4d1-932a-42de-ad6c-4738ca9bef00 None admin] RBAC: Authorizing `identity:get_project(project_id=06615629856547fd9da00a208187c681)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:43.591327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea55b4d1-932a-42de-ad6c-4738ca9bef00 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:43.594797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 105/208] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:43 2022] GET /identity/v3/projects/06615629856547fd9da00a208187c681 => generated 302 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:43.608497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea55b4d1-932a-42de-ad6c-4738ca9bef00 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:43.694736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab4b7dc-54af-4bdf-89a2-cd21152e0b96 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:43.729664 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:43.732331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:43.740650 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc50bac-0029-439a-bf82-d78acd8d581a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:43.741306 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc50bac-0029-439a-bf82-d78acd8d581a None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:43.741590 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc50bac-0029-439a-bf82-d78acd8d581a None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:43.741868 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc50bac-0029-439a-bf82-d78acd8d581a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:43.743074 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc50bac-0029-439a-bf82-d78acd8d581a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:43.744684 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc50bac-0029-439a-bf82-d78acd8d581a None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:43.746442 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 104/209] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:43:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:43.747868 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0833422-bf05-4e46-bd7c-4b862320ee2b None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:43.748038 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0833422-bf05-4e46-bd7c-4b862320ee2b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:43.748328 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0833422-bf05-4e46-bd7c-4b862320ee2b None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:43.749860 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 105/210] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:43.750409 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0833422-bf05-4e46-bd7c-4b862320ee2b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:43.771286 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea55b4d1-932a-42de-ad6c-4738ca9bef00 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:43.826712 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874aea34-8019-4b15-821e-7c15c057700b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:43.827736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874aea34-8019-4b15-821e-7c15c057700b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:43.828090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874aea34-8019-4b15-821e-7c15c057700b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:43.828455 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874aea34-8019-4b15-821e-7c15c057700b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:43.829299 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874aea34-8019-4b15-821e-7c15c057700b None admin] RBAC: Adding query filter params (role.id=8a6c7a6c7d154f3db6d2639609d60224, scope.project.id=06615629856547fd9da00a208187c681, user.id=74ebd9bcd208442bbf14f5d46a426f42) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:43.829700 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874aea34-8019-4b15-821e-7c15c057700b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:43.831579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874aea34-8019-4b15-821e-7c15c057700b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:43.839282 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 106/211] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:43:43 2022] GET /identity/v3/role_assignments?user.id=74ebd9bcd208442bbf14f5d46a426f42&scope.project.id=06615629856547fd9da00a208187c681&role.id=8a6c7a6c7d154f3db6d2639609d60224 => generated 608 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:43.840285 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874aea34-8019-4b15-821e-7c15c057700b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:43.916329 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0833422-bf05-4e46-bd7c-4b862320ee2b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:43.958623 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5861f4e8-6103-4954-8427-c1aa429bf697 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:43.959219 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5861f4e8-6103-4954-8427-c1aa429bf697 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:43.959623 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5861f4e8-6103-4954-8427-c1aa429bf697 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:43.959782 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5861f4e8-6103-4954-8427-c1aa429bf697 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:43.979007 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5861f4e8-6103-4954-8427-c1aa429bf697 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:43.988086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-5861f4e8-6103-4954-8427-c1aa429bf697 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:43.988086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:43.988086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:43.988086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:43.988086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:43.988086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:43.988086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:43.988086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:43.988086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:43.988086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:43.988086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:43.988086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:43.988086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:43.988086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:43.988086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:43.988086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:43.990159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:43.990159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:43.990159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:43.990159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:43.990159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:43.990159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:43.990159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:43.990159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:43.990159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:43.990159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:43.990159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:43.990159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:43.990159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:43.990159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:43.990159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:43.990159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:43.992616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:43.992616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:43.992616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:43.992616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:43.992616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:43.992616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:43.992616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:43.992616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:43.992616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:43.992616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:43.992616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:43.992616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:43.992616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 106/212] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:43 2022] GET /identity/v3/roles/member => generated 84 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:43.992616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e6ebad-8e72-48dc-8dd9-b41efedc2761 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:43.994030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e6ebad-8e72-48dc-8dd9-b41efedc2761 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:43.994030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e6ebad-8e72-48dc-8dd9-b41efedc2761 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:44.015281 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874aea34-8019-4b15-821e-7c15c057700b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:44.024968 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-06e6ebad-8e72-48dc-8dd9-b41efedc2761 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:44.027592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-06e6ebad-8e72-48dc-8dd9-b41efedc2761 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:44.056905 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054dfce9-49b2-4068-a903-cef0f735a410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:44.058100 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054dfce9-49b2-4068-a903-cef0f735a410 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:44.058413 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054dfce9-49b2-4068-a903-cef0f735a410 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:44.058658 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054dfce9-49b2-4068-a903-cef0f735a410 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:44.059363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054dfce9-49b2-4068-a903-cef0f735a410 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:44.060882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054dfce9-49b2-4068-a903-cef0f735a410 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:44.083124 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:44.092308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 107/213] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:43:43 2022] POST /identity/v3/users => generated 293 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:43:44.093256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054dfce9-49b2-4068-a903-cef0f735a410 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:44.140491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:44.183271 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 107/214] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:44.264687 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054dfce9-49b2-4068-a903-cef0f735a410 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:44.310498 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e605410-47dc-4415-a15f-68dbd43561eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:44.311184 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e605410-47dc-4415-a15f-68dbd43561eb None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:44.311467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e605410-47dc-4415-a15f-68dbd43561eb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:44.311741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e605410-47dc-4415-a15f-68dbd43561eb None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:44.312438 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e605410-47dc-4415-a15f-68dbd43561eb None admin] RBAC: Adding query filter params (name=member) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:44.312748 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e605410-47dc-4415-a15f-68dbd43561eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:44.314210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e605410-47dc-4415-a15f-68dbd43561eb None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:44.325204 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 108/215] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:43:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:44.334897 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e6ebad-8e72-48dc-8dd9-b41efedc2761 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:44.476394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e6ebad-8e72-48dc-8dd9-b41efedc2761 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:44.536300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c6525a-3e76-4790-9aad-c768533c5be0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:44.537722 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c6525a-3e76-4790-9aad-c768533c5be0 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:44.538436 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c6525a-3e76-4790-9aad-c768533c5be0 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:44.539171 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c6525a-3e76-4790-9aad-c768533c5be0 None admin] PATH_INFO: `/v3/users/4cc1646e207c4a4988c843b18a8a3e8a` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:44.566731 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c6525a-3e76-4790-9aad-c768533c5be0 None admin] RBAC: Authorizing `identity:get_user(user_id=4cc1646e207c4a4988c843b18a8a3e8a)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:44.569593 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c6525a-3e76-4790-9aad-c768533c5be0 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:44.572977 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 108/216] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:44 2022] GET /identity/v3/users/4cc1646e207c4a4988c843b18a8a3e8a => generated 293 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:44.582497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e605410-47dc-4415-a15f-68dbd43561eb None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:44.685334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:44.691772 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c6525a-3e76-4790-9aad-c768533c5be0 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:44.744437 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e605410-47dc-4415-a15f-68dbd43561eb None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:44.752589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:44.798539 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec073c04-1df0-4911-ac22-c6ae40bdc38a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:44.799396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec073c04-1df0-4911-ac22-c6ae40bdc38a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:44.799824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec073c04-1df0-4911-ac22-c6ae40bdc38a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:44.800145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec073c04-1df0-4911-ac22-c6ae40bdc38a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:44.801058 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec073c04-1df0-4911-ac22-c6ae40bdc38a None admin] RBAC: Adding query filter params (role.id=8a6c7a6c7d154f3db6d2639609d60224, user.id=4cc1646e207c4a4988c843b18a8a3e8a, scope.system=all) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:44.801477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec073c04-1df0-4911-ac22-c6ae40bdc38a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:44.803193 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec073c04-1df0-4911-ac22-c6ae40bdc38a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:44.810470 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 109/217] 10.222.0.25 () {56 vars in 1308 bytes} [Sun Oct 16 00:43:44 2022] GET /identity/v3/role_assignments?user.id=4cc1646e207c4a4988c843b18a8a3e8a&scope.system=all&role.id=8a6c7a6c7d154f3db6d2639609d60224 => generated 226 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:44.836854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c6525a-3e76-4790-9aad-c768533c5be0 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:44.881237 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7097d1-15c4-4227-8a89-6461f9119209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:44.882081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7097d1-15c4-4227-8a89-6461f9119209 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:44.882469 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7097d1-15c4-4227-8a89-6461f9119209 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:44.882850 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7097d1-15c4-4227-8a89-6461f9119209 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:44.896060 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7097d1-15c4-4227-8a89-6461f9119209 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:44.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-0a7097d1-15c4-4227-8a89-6461f9119209 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 16 00:43:44.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:44.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:44.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:44.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:44.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:44.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:44.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:44.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:44.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:44.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:44.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:44.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:44.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:44.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:44.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:44.904055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:44.904055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:44.904055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:44.904055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:44.904055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:44.904055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:44.904055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:44.904055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:44.904055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:44.904055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:44.904055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:44.904055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:44.904055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:44.904055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:44.904055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:44.904055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:44.905899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:44.905899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:44.905899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:44.905899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:44.905899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:44.905899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:44.905899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:44.905899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:44.905899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:44.905899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:44.905899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 16 00:43:44.905899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:44.905899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 109/218] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:44 2022] GET /identity/v3/roles/reader => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:44.915701 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec073c04-1df0-4911-ac22-c6ae40bdc38a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:45.039465 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec073c04-1df0-4911-ac22-c6ae40bdc38a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:45.067160 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:45.081130 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52982f0e-d897-4ba0-80c9-bcfda49ed2a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:45.081894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52982f0e-d897-4ba0-80c9-bcfda49ed2a0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:45.082271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52982f0e-d897-4ba0-80c9-bcfda49ed2a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:45.082644 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52982f0e-d897-4ba0-80c9-bcfda49ed2a0 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:45.083452 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52982f0e-d897-4ba0-80c9-bcfda49ed2a0 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:45.083888 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52982f0e-d897-4ba0-80c9-bcfda49ed2a0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:45.085421 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52982f0e-d897-4ba0-80c9-bcfda49ed2a0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:45.102826 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 110/219] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:43:44 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:45.108609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7097d1-15c4-4227-8a89-6461f9119209 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:45.228611 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7097d1-15c4-4227-8a89-6461f9119209 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:45.231246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:45.272355 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcad3924-687d-466a-a4ce-deace4eeaa8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:45.273022 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcad3924-687d-466a-a4ce-deace4eeaa8e None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:45.273306 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcad3924-687d-466a-a4ce-deace4eeaa8e None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:45.273585 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcad3924-687d-466a-a4ce-deace4eeaa8e None admin] PATH_INFO: `/v3/users/7beaa53b40cd4f2e925660bc4bfb61a7` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:45.274822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcad3924-687d-466a-a4ce-deace4eeaa8e None admin] RBAC: Authorizing `identity:get_user(user_id=7beaa53b40cd4f2e925660bc4bfb61a7)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:45.276410 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcad3924-687d-466a-a4ce-deace4eeaa8e None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:45.278393 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 110/220] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:45 2022] GET /identity/v3/users/7beaa53b40cd4f2e925660bc4bfb61a7 => generated 297 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:45.287640 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52982f0e-d897-4ba0-80c9-bcfda49ed2a0 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:45.421069 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52982f0e-d897-4ba0-80c9-bcfda49ed2a0 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:45.468738 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea91e6b-06aa-4643-90bb-5c72ecb99c33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:45.469364 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea91e6b-06aa-4643-90bb-5c72ecb99c33 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:45.469683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea91e6b-06aa-4643-90bb-5c72ecb99c33 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:45.469958 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea91e6b-06aa-4643-90bb-5c72ecb99c33 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:45.543596 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:45.559583 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea91e6b-06aa-4643-90bb-5c72ecb99c33 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:45.561079 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea91e6b-06aa-4643-90bb-5c72ecb99c33 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:45.673401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 111/221] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:43:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:43:45.679322 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcad3924-687d-466a-a4ce-deace4eeaa8e None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:45.790436 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcad3924-687d-466a-a4ce-deace4eeaa8e None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:45.828631 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda71fd7-ced7-4489-8d36-4e461f30739d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:45.829291 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda71fd7-ced7-4489-8d36-4e461f30739d None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:45.829578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda71fd7-ced7-4489-8d36-4e461f30739d None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:45.829853 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda71fd7-ced7-4489-8d36-4e461f30739d None admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:45.831353 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda71fd7-ced7-4489-8d36-4e461f30739d None admin] RBAC: Authorizing `identity:get_project(project_id=06615629856547fd9da00a208187c681)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:45.832839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda71fd7-ced7-4489-8d36-4e461f30739d None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:45.834533 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 111/222] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:43:45 2022] GET /identity/v3/projects/06615629856547fd9da00a208187c681 => generated 302 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:45.844890 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea91e6b-06aa-4643-90bb-5c72ecb99c33 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:45.997322 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:46.016490 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea91e6b-06aa-4643-90bb-5c72ecb99c33 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:46.056032 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7e9a0d-9a91-4340-9c5f-4d6ea0a96f66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:46.056662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7e9a0d-9a91-4340-9c5f-4d6ea0a96f66 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:46.057531 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7e9a0d-9a91-4340-9c5f-4d6ea0a96f66 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:46.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7e9a0d-9a91-4340-9c5f-4d6ea0a96f66 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:46.058460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7e9a0d-9a91-4340-9c5f-4d6ea0a96f66 None admin] RBAC: Adding query filter params (role.id=1ac2f92c6eb34313ac9d9dfd4476663c, scope.project.id=06615629856547fd9da00a208187c681, user.id=7beaa53b40cd4f2e925660bc4bfb61a7) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:46.058774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7e9a0d-9a91-4340-9c5f-4d6ea0a96f66 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:46.060265 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7e9a0d-9a91-4340-9c5f-4d6ea0a96f66 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:46.066892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 112/223] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:43:45 2022] GET /identity/v3/role_assignments?user.id=7beaa53b40cd4f2e925660bc4bfb61a7&scope.project.id=06615629856547fd9da00a208187c681&role.id=1ac2f92c6eb34313ac9d9dfd4476663c => generated 608 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:46.893723 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e33965-c487-4b4a-a59f-8c7ce7e3e18d None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:46.894302 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e33965-c487-4b4a-a59f-8c7ce7e3e18d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:46.894302 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e33965-c487-4b4a-a59f-8c7ce7e3e18d None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:46.896032 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 112/224] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:46.902176 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b66f6c2-0848-474d-9cf7-ed4f61fd32d1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:46.902589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b66f6c2-0848-474d-9cf7-ed4f61fd32d1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:46.902762 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b66f6c2-0848-474d-9cf7-ed4f61fd32d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:46.916929 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c6a421-2500-4242-95bc-6c2839d9aba7 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:46.917262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c6a421-2500-4242-95bc-6c2839d9aba7 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:46.917477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c6a421-2500-4242-95bc-6c2839d9aba7 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:46.919136 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 113/225] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:43:46.924857 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd076622-c95f-49c7-9b0b-65b810afc75a None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:46.925162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd076622-c95f-49c7-9b0b-65b810afc75a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:46.925412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd076622-c95f-49c7-9b0b-65b810afc75a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:46.929009 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4b66f6c2-0848-474d-9cf7-ed4f61fd32d1 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:46.930819 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4b66f6c2-0848-474d-9cf7-ed4f61fd32d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:46.945814 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-cd076622-c95f-49c7-9b0b-65b810afc75a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:46.947118 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-cd076622-c95f-49c7-9b0b-65b810afc75a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:46.985323 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:47.038433 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 113/226] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:43:47.039180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 114/227] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:43:47.310164 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b66f6c2-0848-474d-9cf7-ed4f61fd32d1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:47.384887 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5b78c4-2acc-4a2c-9df2-d60850afe657 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:47.385699 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5b78c4-2acc-4a2c-9df2-d60850afe657 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:47.386053 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5b78c4-2acc-4a2c-9df2-d60850afe657 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:47.387820 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 115/228] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:43:47.393529 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2d97db-5323-4ea5-b7f5-e4864f947ea4 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:47.393842 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2d97db-5323-4ea5-b7f5-e4864f947ea4 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:47.394054 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2d97db-5323-4ea5-b7f5-e4864f947ea4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:47.409671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:47.418332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-1b2d97db-5323-4ea5-b7f5-e4864f947ea4 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:47.419597 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-1b2d97db-5323-4ea5-b7f5-e4864f947ea4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:47.440100 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b66f6c2-0848-474d-9cf7-ed4f61fd32d1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:47.480304 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac925c9-7872-4240-9845-b0320589274e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:47.481113 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac925c9-7872-4240-9845-b0320589274e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:47.481381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac925c9-7872-4240-9845-b0320589274e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:47.481657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac925c9-7872-4240-9845-b0320589274e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:47.494081 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac925c9-7872-4240-9845-b0320589274e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:47.500337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-7ac925c9-7872-4240-9845-b0320589274e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 16 00:43:47.500337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:47.500337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:47.500337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:47.500337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:47.500337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:47.500337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:47.500337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:47.500337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:47.500337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:47.500337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:47.500337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:47.500337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:47.500337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:47.500337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:47.500337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:47.501327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:47.501327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:47.501327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:47.501327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:47.501327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:47.501327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:47.501327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:47.501327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:47.501327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:47.501327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:47.501327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:47.501327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:47.501327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:47.501327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:47.501327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:47.501327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:47.502161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:47.502161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:47.502161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:47.502161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:47.502161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:47.502161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:47.502161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:47.502161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:47.502161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:47.502161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:47.502161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 16 00:43:47.502161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:47.502161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 114/229] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:47 2022] GET /identity/v3/roles/reader => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:43:47.505668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 116/230] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:47.508990 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac925c9-7872-4240-9845-b0320589274e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:47.529591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2d97db-5323-4ea5-b7f5-e4864f947ea4 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:47.651846 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac925c9-7872-4240-9845-b0320589274e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:47.668971 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:47.675471 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2d97db-5323-4ea5-b7f5-e4864f947ea4 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:47.688821 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b28b36-a634-4c52-a96b-4095d4617337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:47.689438 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b28b36-a634-4c52-a96b-4095d4617337 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:47.689721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b28b36-a634-4c52-a96b-4095d4617337 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:47.689997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b28b36-a634-4c52-a96b-4095d4617337 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:47.690946 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b28b36-a634-4c52-a96b-4095d4617337 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:47.691080 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b28b36-a634-4c52-a96b-4095d4617337 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:47.692414 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b28b36-a634-4c52-a96b-4095d4617337 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:47.701298 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 115/231] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:43:47 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:47.710259 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b28b36-a634-4c52-a96b-4095d4617337 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:47.715164 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:47.726404 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67000532-55d4-4065-9301-5f63453fad64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:47.727140 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67000532-55d4-4065-9301-5f63453fad64 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:47.727355 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67000532-55d4-4065-9301-5f63453fad64 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:47.727624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67000532-55d4-4065-9301-5f63453fad64 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:47.729123 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67000532-55d4-4065-9301-5f63453fad64 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:47.730730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67000532-55d4-4065-9301-5f63453fad64 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:47.732575 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 117/232] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:43:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:47.737751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67000532-55d4-4065-9301-5f63453fad64 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:47.842100 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b28b36-a634-4c52-a96b-4095d4617337 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:47.861125 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67000532-55d4-4065-9301-5f63453fad64 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:47.875727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33e8420-6ef3-448b-8801-3bb96e0528a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:47.876359 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33e8420-6ef3-448b-8801-3bb96e0528a3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:47.876637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33e8420-6ef3-448b-8801-3bb96e0528a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:47.876912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33e8420-6ef3-448b-8801-3bb96e0528a3 None admin] PATH_INFO: `/v3/users/40fd3e0b8a9a4c8eb5b9e2167be54acf` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:47.892545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33e8420-6ef3-448b-8801-3bb96e0528a3 None admin] RBAC: Authorizing `identity:get_user(user_id=40fd3e0b8a9a4c8eb5b9e2167be54acf)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:47.894631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33e8420-6ef3-448b-8801-3bb96e0528a3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:47.896701 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 116/233] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:47 2022] GET /identity/v3/users/40fd3e0b8a9a4c8eb5b9e2167be54acf => generated 293 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:47.898178 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ecd881-65e7-44a3-824c-cf90c8cfbe53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:47.898806 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ecd881-65e7-44a3-824c-cf90c8cfbe53 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:47.899112 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ecd881-65e7-44a3-824c-cf90c8cfbe53 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:47.899388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ecd881-65e7-44a3-824c-cf90c8cfbe53 None admin] PATH_INFO: `/v3/groups` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:47.899754 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33e8420-6ef3-448b-8801-3bb96e0528a3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:47.900083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ecd881-65e7-44a3-824c-cf90c8cfbe53 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:47.902514 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ecd881-65e7-44a3-824c-cf90c8cfbe53 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:47.920569 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 118/234] 10.222.0.25 () {60 vars in 1134 bytes} [Sun Oct 16 00:43:47 2022] POST /identity/v3/groups => generated 229 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:43:47.926814 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ecd881-65e7-44a3-824c-cf90c8cfbe53 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:47.966795 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:47.983421 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:48.042812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33e8420-6ef3-448b-8801-3bb96e0528a3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:48.080332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ecd881-65e7-44a3-824c-cf90c8cfbe53 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:48.092709 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24af02a6-e580-4bcc-a50d-5fb5b2ade3ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:48.093367 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24af02a6-e580-4bcc-a50d-5fb5b2ade3ca None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:48.093649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24af02a6-e580-4bcc-a50d-5fb5b2ade3ca None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:48.093923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24af02a6-e580-4bcc-a50d-5fb5b2ade3ca None admin] PATH_INFO: `/v3/roles/member` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:48.106255 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24af02a6-e580-4bcc-a50d-5fb5b2ade3ca None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:48.111289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-24af02a6-e580-4bcc-a50d-5fb5b2ade3ca None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:48.111289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:48.111289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:48.111289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:48.111289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:48.111289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:48.111289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:48.111289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:48.111289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:48.111289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:48.111289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:48.111289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:48.111289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:48.111289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:48.111289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:48.111289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:48.113035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:48.113035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:48.113035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:48.113035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:48.113035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:48.113035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:48.113035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:48.113035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:48.113035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:48.113035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:48.113035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:48.113035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:48.113035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:48.113035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:48.113035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:48.113035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:48.114483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:48.114483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:48.114483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:48.114483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:48.114483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:48.114483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:48.114483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:48.114483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:48.114483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:48.114483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:48.114483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:48.114483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:48.114483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 117/235] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:48.118999 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24af02a6-e580-4bcc-a50d-5fb5b2ade3ca None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:48.144056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5401d20c-a767-49ec-a33c-9b9b44bb739c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:48.144847 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5401d20c-a767-49ec-a33c-9b9b44bb739c None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:48.146904 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5401d20c-a767-49ec-a33c-9b9b44bb739c None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:48.146904 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5401d20c-a767-49ec-a33c-9b9b44bb739c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:48.147193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5401d20c-a767-49ec-a33c-9b9b44bb739c None admin] RBAC: Adding query filter params (role.id=1ac2f92c6eb34313ac9d9dfd4476663c, user.id=40fd3e0b8a9a4c8eb5b9e2167be54acf, scope.system=all) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:48.147620 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5401d20c-a767-49ec-a33c-9b9b44bb739c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:48.149990 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5401d20c-a767-49ec-a33c-9b9b44bb739c None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:48.157708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 119/236] 10.222.0.25 () {56 vars in 1308 bytes} [Sun Oct 16 00:43:47 2022] GET /identity/v3/role_assignments?user.id=40fd3e0b8a9a4c8eb5b9e2167be54acf&scope.system=all&role.id=1ac2f92c6eb34313ac9d9dfd4476663c => generated 226 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:48.268923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24af02a6-e580-4bcc-a50d-5fb5b2ade3ca None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:48.294424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:48.315722 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7f45a3-c6a6-4350-bf05-8921f431a64c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:48.316432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7f45a3-c6a6-4350-bf05-8921f431a64c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:48.316745 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7f45a3-c6a6-4350-bf05-8921f431a64c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:48.316988 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7f45a3-c6a6-4350-bf05-8921f431a64c None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:48.317678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7f45a3-c6a6-4350-bf05-8921f431a64c None admin] RBAC: Adding query filter params (name=member) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:48.317989 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7f45a3-c6a6-4350-bf05-8921f431a64c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:48.319486 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7f45a3-c6a6-4350-bf05-8921f431a64c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:48.333015 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 118/237] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:43:48 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:48.342165 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5401d20c-a767-49ec-a33c-9b9b44bb739c None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:48.453391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:48.465633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5401d20c-a767-49ec-a33c-9b9b44bb739c None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:48.501730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fecb342-f72f-42c8-bd24-48a36d989aea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:48.502378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fecb342-f72f-42c8-bd24-48a36d989aea None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:48.502662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fecb342-f72f-42c8-bd24-48a36d989aea None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:48.502998 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fecb342-f72f-42c8-bd24-48a36d989aea None admin] PATH_INFO: `/v3/users/4cc1646e207c4a4988c843b18a8a3e8a` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:48.504145 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fecb342-f72f-42c8-bd24-48a36d989aea None admin] RBAC: Authorizing `identity:get_user(user_id=4cc1646e207c4a4988c843b18a8a3e8a)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:48.505615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fecb342-f72f-42c8-bd24-48a36d989aea None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:48.507475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 120/238] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:48 2022] GET /identity/v3/users/4cc1646e207c4a4988c843b18a8a3e8a => generated 293 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:48.513364 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7f45a3-c6a6-4350-bf05-8921f431a64c None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:48.632299 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7f45a3-c6a6-4350-bf05-8921f431a64c None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:48.668419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46ce7d7-d2fd-451e-9f3c-c9c13c3c56f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:48.669086 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46ce7d7-d2fd-451e-9f3c-c9c13c3c56f2 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:48.669370 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46ce7d7-d2fd-451e-9f3c-c9c13c3c56f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:48.669652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46ce7d7-d2fd-451e-9f3c-c9c13c3c56f2 None admin] PATH_INFO: `/v3/system/users/4cc1646e207c4a4988c843b18a8a3e8a/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:48.671139 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f46ce7d7-d2fd-451e-9f3c-c9c13c3c56f2 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=4cc1646e207c4a4988c843b18a8a3e8a, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:48.672550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f46ce7d7-d2fd-451e-9f3c-c9c13c3c56f2 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:48.686064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 119/239] 10.222.0.25 () {58 vars in 1334 bytes} [Sun Oct 16 00:43:48 2022] PUT /identity/v3/system/users/4cc1646e207c4a4988c843b18a8a3e8a/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:43:50.459004 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa63673-f9b6-49f9-8efd-3eef10b75401 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:50.459796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa63673-f9b6-49f9-8efd-3eef10b75401 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:50.459929 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa63673-f9b6-49f9-8efd-3eef10b75401 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:50.461704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 121/240] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:50.467685 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d1eafe-3c15-4848-8d1b-49693243c695 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:50.468232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d1eafe-3c15-4848-8d1b-49693243c695 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:50.468232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d1eafe-3c15-4848-8d1b-49693243c695 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:50.468635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d48803-e22c-4e01-87a7-01a75a40ef1a None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:50.469050 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d48803-e22c-4e01-87a7-01a75a40ef1a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:50.469467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d48803-e22c-4e01-87a7-01a75a40ef1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:50.469618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 120/241] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:43:50.481634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24133e1e-deed-4cf5-a736-87b54eddf3b0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:50.482063 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24133e1e-deed-4cf5-a736-87b54eddf3b0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:50.482462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24133e1e-deed-4cf5-a736-87b54eddf3b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:50.492956 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-91d48803-e22c-4e01-87a7-01a75a40ef1a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:50.494263 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-91d48803-e22c-4e01-87a7-01a75a40ef1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:50.504106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-24133e1e-deed-4cf5-a736-87b54eddf3b0 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:50.505508 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-24133e1e-deed-4cf5-a736-87b54eddf3b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:50.560420 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:50.582646 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 122/242] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:43:50.596140 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 121/243] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:43:51.067185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d48803-e22c-4e01-87a7-01a75a40ef1a None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:51.080140 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24133e1e-deed-4cf5-a736-87b54eddf3b0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:51.193848 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:51.210917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d48803-e22c-4e01-87a7-01a75a40ef1a None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:51.233261 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24133e1e-deed-4cf5-a736-87b54eddf3b0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:51.252571 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5dc58b-420c-4971-b2b3-f350f3811ac8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:51.253371 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5dc58b-420c-4971-b2b3-f350f3811ac8 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:51.253742 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5dc58b-420c-4971-b2b3-f350f3811ac8 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:51.254128 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5dc58b-420c-4971-b2b3-f350f3811ac8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:51.255519 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5dc58b-420c-4971-b2b3-f350f3811ac8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:51.257080 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5dc58b-420c-4971-b2b3-f350f3811ac8 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:51.258704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 123/244] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:43:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:51.259937 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0761845-5921-4c7c-a663-b1bc08b11d85 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:51.261923 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0761845-5921-4c7c-a663-b1bc08b11d85 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:51.261923 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0761845-5921-4c7c-a663-b1bc08b11d85 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:51.264318 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 124/245] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:51 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:51.268026 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0761845-5921-4c7c-a663-b1bc08b11d85 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:51.278657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b254e5f-2e8e-4dce-9403-0f0fcf7aaa00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:51.279355 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b254e5f-2e8e-4dce-9403-0f0fcf7aaa00 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:51.279638 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b254e5f-2e8e-4dce-9403-0f0fcf7aaa00 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:51.279916 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b254e5f-2e8e-4dce-9403-0f0fcf7aaa00 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:51.292299 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b254e5f-2e8e-4dce-9403-0f0fcf7aaa00 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:51.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-9b254e5f-2e8e-4dce-9403-0f0fcf7aaa00 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 16 00:43:51.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:51.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:51.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:51.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:51.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:51.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:51.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:51.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:51.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:51.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:51.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:51.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:51.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:51.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:51.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:51.299581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:51.299581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:51.299581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:51.299581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:51.299581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:51.299581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:51.299581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:51.299581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:51.299581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:51.299581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:51.299581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:51.299581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:51.299581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:51.299581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:51.299581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:51.299581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:51.304713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:51.304713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:51.304713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:51.304713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:51.304713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:51.304713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:51.304713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:51.304713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:51.304713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:51.304713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:51.304713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 16 00:43:51.304713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:51.304713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 122/246] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:51 2022] GET /identity/v3/roles/reader => generated 84 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:51.304713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e85a387-0a42-4a1d-8745-c73ea1f690b7 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:51.306279 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e85a387-0a42-4a1d-8745-c73ea1f690b7 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:51.306279 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e85a387-0a42-4a1d-8745-c73ea1f690b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:51.326086 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4e85a387-0a42-4a1d-8745-c73ea1f690b7 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:51.327514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4e85a387-0a42-4a1d-8745-c73ea1f690b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:51.365227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:51.396173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0761845-5921-4c7c-a663-b1bc08b11d85 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:51.425269 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 123/247] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:43:51.426170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e85a387-0a42-4a1d-8745-c73ea1f690b7 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:51.459484 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e6fdc4-ed1e-4733-a312-608858ca013d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:51.460538 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e6fdc4-ed1e-4733-a312-608858ca013d None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:51.460908 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e6fdc4-ed1e-4733-a312-608858ca013d None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:51.461318 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e6fdc4-ed1e-4733-a312-608858ca013d None admin] PATH_INFO: `/v3/groups` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:51.462409 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e6fdc4-ed1e-4733-a312-608858ca013d None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:51.465884 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e6fdc4-ed1e-4733-a312-608858ca013d None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:51.490954 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 125/248] 10.222.0.25 () {60 vars in 1134 bytes} [Sun Oct 16 00:43:51 2022] POST /identity/v3/groups => generated 226 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:43:51.581251 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e85a387-0a42-4a1d-8745-c73ea1f690b7 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:51.625229 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976cc9dd-e77c-49af-af93-714345d73e1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:51.626046 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976cc9dd-e77c-49af-af93-714345d73e1b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:51.626457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976cc9dd-e77c-49af-af93-714345d73e1b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:51.626904 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976cc9dd-e77c-49af-af93-714345d73e1b None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:51.627731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976cc9dd-e77c-49af-af93-714345d73e1b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:51.628176 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976cc9dd-e77c-49af-af93-714345d73e1b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:51.629748 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976cc9dd-e77c-49af-af93-714345d73e1b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:51.641236 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 124/249] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:43:51 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:51.647267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e6fdc4-ed1e-4733-a312-608858ca013d None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:51.669653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:51.701781 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976cc9dd-e77c-49af-af93-714345d73e1b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:51.739038 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:51.799553 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e6fdc4-ed1e-4733-a312-608858ca013d None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:51.856367 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976cc9dd-e77c-49af-af93-714345d73e1b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:51.860946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12edc986-f250-4fc1-88c3-987b0856b618 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:51.861774 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12edc986-f250-4fc1-88c3-987b0856b618 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:51.862175 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12edc986-f250-4fc1-88c3-987b0856b618 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:51.862584 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12edc986-f250-4fc1-88c3-987b0856b618 None admin] PATH_INFO: `/v3/users/40fd3e0b8a9a4c8eb5b9e2167be54acf` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:51.864015 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12edc986-f250-4fc1-88c3-987b0856b618 None admin] RBAC: Authorizing `identity:get_user(user_id=40fd3e0b8a9a4c8eb5b9e2167be54acf)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:51.866258 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12edc986-f250-4fc1-88c3-987b0856b618 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:51.868050 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 126/250] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:51 2022] GET /identity/v3/users/40fd3e0b8a9a4c8eb5b9e2167be54acf => generated 293 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:51.875079 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12edc986-f250-4fc1-88c3-987b0856b618 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:51.911908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27fe465-33ef-4653-a84e-c0e89fb9020b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:51.913491 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27fe465-33ef-4653-a84e-c0e89fb9020b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:51.914252 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27fe465-33ef-4653-a84e-c0e89fb9020b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:51.915020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27fe465-33ef-4653-a84e-c0e89fb9020b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:51.939912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e27fe465-33ef-4653-a84e-c0e89fb9020b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:51.950454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-e27fe465-33ef-4653-a84e-c0e89fb9020b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:51.950454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:51.950454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:51.950454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:51.950454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:51.950454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:51.950454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:51.950454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:51.950454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:51.950454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:51.950454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:51.950454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:51.950454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:51.950454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:51.950454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:51.950454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:51.952235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:51.952235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:51.952235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:51.952235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:51.952235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:51.952235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:51.952235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:51.952235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:51.952235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:51.952235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:51.952235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:51.952235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:51.952235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:51.952235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:51.952235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:51.952235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:51.953782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:51.953782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:51.953782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:51.953782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:51.953782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:51.953782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:51.953782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:51.953782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:51.953782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:51.953782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:51.953782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:51.953782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:51.953782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 125/251] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:43:51.962678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27fe465-33ef-4653-a84e-c0e89fb9020b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:52.061852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:52.097575 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12edc986-f250-4fc1-88c3-987b0856b618 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:52.148631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:52.157123 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3940704-8990-4fa0-bbfb-7cc44ea9180a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:52.158064 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3940704-8990-4fa0-bbfb-7cc44ea9180a None admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:52.158499 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3940704-8990-4fa0-bbfb-7cc44ea9180a None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:52.158965 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3940704-8990-4fa0-bbfb-7cc44ea9180a None admin] PATH_INFO: `/v3/system/users/40fd3e0b8a9a4c8eb5b9e2167be54acf/roles/1ac2f92c6eb34313ac9d9dfd4476663c` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:52.161187 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3940704-8990-4fa0-bbfb-7cc44ea9180a None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=40fd3e0b8a9a4c8eb5b9e2167be54acf, role_id=1ac2f92c6eb34313ac9d9dfd4476663c)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:52.163240 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3940704-8990-4fa0-bbfb-7cc44ea9180a None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:52.175125 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 127/252] 10.222.0.25 () {58 vars in 1334 bytes} [Sun Oct 16 00:43:51 2022] PUT /identity/v3/system/users/40fd3e0b8a9a4c8eb5b9e2167be54acf/roles/1ac2f92c6eb34313ac9d9dfd4476663c => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:43:52.202803 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27fe465-33ef-4653-a84e-c0e89fb9020b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:52.291374 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba15dd1-6199-41f4-8db6-7453417ce8de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:52.292404 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba15dd1-6199-41f4-8db6-7453417ce8de None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:52.292885 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba15dd1-6199-41f4-8db6-7453417ce8de None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:52.293340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba15dd1-6199-41f4-8db6-7453417ce8de None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:52.294525 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba15dd1-6199-41f4-8db6-7453417ce8de None admin] RBAC: Adding query filter params (name=member) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:52.295039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba15dd1-6199-41f4-8db6-7453417ce8de None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:52.297996 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba15dd1-6199-41f4-8db6-7453417ce8de None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:52.315320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 126/253] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:43:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:52.324897 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3940704-8990-4fa0-bbfb-7cc44ea9180a None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:52.520201 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3940704-8990-4fa0-bbfb-7cc44ea9180a None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:52.566319 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eee595b-5fb8-4272-a2b2-4c90aafc5708 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:52.567012 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eee595b-5fb8-4272-a2b2-4c90aafc5708 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:52.567299 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eee595b-5fb8-4272-a2b2-4c90aafc5708 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:52.567625 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eee595b-5fb8-4272-a2b2-4c90aafc5708 None admin] PATH_INFO: `/v3/users/4cc1646e207c4a4988c843b18a8a3e8a` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:52.568770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eee595b-5fb8-4272-a2b2-4c90aafc5708 None admin] RBAC: Authorizing `identity:get_user(user_id=4cc1646e207c4a4988c843b18a8a3e8a)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:52.570254 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eee595b-5fb8-4272-a2b2-4c90aafc5708 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:52.571898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 128/254] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:52 2022] GET /identity/v3/users/4cc1646e207c4a4988c843b18a8a3e8a => generated 293 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:52.578646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba15dd1-6199-41f4-8db6-7453417ce8de None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:52.746373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba15dd1-6199-41f4-8db6-7453417ce8de None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:52.819757 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6537c55-8240-41c4-96d9-fabfdc3b5567 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:52.820682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6537c55-8240-41c4-96d9-fabfdc3b5567 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:52.821004 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6537c55-8240-41c4-96d9-fabfdc3b5567 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:52.821291 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6537c55-8240-41c4-96d9-fabfdc3b5567 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:52.822094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6537c55-8240-41c4-96d9-fabfdc3b5567 None admin] RBAC: Adding query filter params (role.id=8a6c7a6c7d154f3db6d2639609d60224, user.id=4cc1646e207c4a4988c843b18a8a3e8a, scope.system=all) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:52.822441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6537c55-8240-41c4-96d9-fabfdc3b5567 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:52.824107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6537c55-8240-41c4-96d9-fabfdc3b5567 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:52.849772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 127/255] 10.222.0.25 () {56 vars in 1308 bytes} [Sun Oct 16 00:43:52 2022] GET /identity/v3/role_assignments?user.id=4cc1646e207c4a4988c843b18a8a3e8a&scope.system=all&role.id=8a6c7a6c7d154f3db6d2639609d60224 => generated 510 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:55.040053 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0056895f-a9c6-4511-82e5-44cfa80e6e47 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:55.040973 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0056895f-a9c6-4511-82e5-44cfa80e6e47 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:55.040973 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0056895f-a9c6-4511-82e5-44cfa80e6e47 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:55.042458 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 129/256] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:55.048620 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfefb49-dc93-4b52-98bd-98006af40f10 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:55.049357 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfefb49-dc93-4b52-98bd-98006af40f10 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:55.049357 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfefb49-dc93-4b52-98bd-98006af40f10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:55.075973 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-8bfefb49-dc93-4b52-98bd-98006af40f10 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:55.077234 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8bfefb49-dc93-4b52-98bd-98006af40f10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:55.089968 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:55.100843 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbe9b8e-503a-4df8-b474-4315ddf50836 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:55.101252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbe9b8e-503a-4df8-b474-4315ddf50836 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:55.101434 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbe9b8e-503a-4df8-b474-4315ddf50836 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:55.102719 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 130/257] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:43:55.109795 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30749271-8e6f-4da7-9767-d29789e70b87 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:55.109795 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30749271-8e6f-4da7-9767-d29789e70b87 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:55.109795 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30749271-8e6f-4da7-9767-d29789e70b87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:55.131621 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-30749271-8e6f-4da7-9767-d29789e70b87 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:55.131621 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-30749271-8e6f-4da7-9767-d29789e70b87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:55.158951 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:55.232035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 128/258] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:55 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 16 00:43:55.281200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 131/259] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:43:55.372521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e580689-b9f3-4a6b-bed2-0ff2a8d9a9aa None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:55.373483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e580689-b9f3-4a6b-bed2-0ff2a8d9a9aa None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:55.374219 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e580689-b9f3-4a6b-bed2-0ff2a8d9a9aa None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:55.377360 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 129/260] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:55 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:55.385277 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2700f6-7a87-4edd-8525-3ae838768cdb None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:55.385654 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2700f6-7a87-4edd-8525-3ae838768cdb None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:55.385932 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2700f6-7a87-4edd-8525-3ae838768cdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:55.416573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-6f2700f6-7a87-4edd-8525-3ae838768cdb None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:55.417911 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6f2700f6-7a87-4edd-8525-3ae838768cdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:55.429889 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 132/261] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:55 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 16 00:43:55.792781 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e580689-b9f3-4a6b-bed2-0ff2a8d9a9aa None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:55.881723 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2700f6-7a87-4edd-8525-3ae838768cdb None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:55.910510 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2700f6-7a87-4edd-8525-3ae838768cdb None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:55.922345 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c328a4-6d27-4acb-b9d0-2ae96242983d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:55.923026 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c328a4-6d27-4acb-b9d0-2ae96242983d admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:55.923317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c328a4-6d27-4acb-b9d0-2ae96242983d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:55.923592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c328a4-6d27-4acb-b9d0-2ae96242983d admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:55.952582 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c328a4-6d27-4acb-b9d0-2ae96242983d admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:55.957839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-b2c328a4-6d27-4acb-b9d0-2ae96242983d admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:55.957839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:55.957839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:55.957839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:55.957839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:55.957839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:55.957839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:55.957839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:55.957839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:55.957839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:55.957839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:55.957839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:55.957839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:55.957839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:55.957839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:55.957839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:55.959528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:55.959528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:55.959528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:55.959528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:55.959528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:55.959528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:55.959528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:55.959528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:55.959528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:55.959528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:55.959528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:55.959528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:55.959528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:55.959528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:55.959528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:55.959528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:55.961124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:55.961124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:55.961124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:55.961124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:55.961124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:55.961124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:55.961124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:55.961124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:55.961124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:55.961124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:55.961124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:55.961124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:55.965363 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 133/262] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Oct 16 00:43:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 84 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:43:55.975344 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c328a4-6d27-4acb-b9d0-2ae96242983d admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:55.976337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e580689-b9f3-4a6b-bed2-0ff2a8d9a9aa None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:56.006113 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c328a4-6d27-4acb-b9d0-2ae96242983d admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:56.016916 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387ebc12-d1e5-4d07-af2e-5634ee6a20a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:56.018213 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387ebc12-d1e5-4d07-af2e-5634ee6a20a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:56.020640 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387ebc12-d1e5-4d07-af2e-5634ee6a20a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:56.021339 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387ebc12-d1e5-4d07-af2e-5634ee6a20a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:56.022303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387ebc12-d1e5-4d07-af2e-5634ee6a20a9 admin admin] RBAC: Adding query filter params (name=member) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:56.024512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387ebc12-d1e5-4d07-af2e-5634ee6a20a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:56.028446 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: WARNING py.warnings [None req-387ebc12-d1e5-4d07-af2e-5634ee6a20a9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 16 00:43:56.028446 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: warnings.warn(msg) Oct 16 00:43:56.028446 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: Oct 16 00:43:56.032313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387ebc12-d1e5-4d07-af2e-5634ee6a20a9 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:56.052933 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78736515-e118-4cd6-bc53-7c2b06ceed7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:56.054339 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78736515-e118-4cd6-bc53-7c2b06ceed7c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:56.054817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78736515-e118-4cd6-bc53-7c2b06ceed7c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:56.059411 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 134/263] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 16 00:43:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:56.060528 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78736515-e118-4cd6-bc53-7c2b06ceed7c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:56.061572 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387ebc12-d1e5-4d07-af2e-5634ee6a20a9 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:56.087597 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387ebc12-d1e5-4d07-af2e-5634ee6a20a9 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:56.091905 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78736515-e118-4cd6-bc53-7c2b06ceed7c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:56.103824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e659e40f-8ebe-4ef9-9c50-d5c1e0d69b8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:56.104964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-78736515-e118-4cd6-bc53-7c2b06ceed7c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 16 00:43:56.104964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:56.104964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:56.104964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:56.104964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:56.104964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:56.104964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:56.104964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:56.104964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:56.104964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:56.104964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:56.104964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:56.104964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:56.104964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:56.104964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:56.104964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:56.105854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:56.105854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:56.105854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:56.105854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:56.105854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:56.105854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:56.105854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:56.105854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:56.105854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:56.105854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:56.105854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:56.105854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:56.105854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:56.105854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:56.105854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:56.105854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:56.106682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:56.106682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:56.106682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:56.106682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:56.106682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:56.106682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:56.106682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:56.106682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:56.106682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:56.106682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:56.106682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 16 00:43:56.106682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:56.107921 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e659e40f-8ebe-4ef9-9c50-d5c1e0d69b8b admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:56.108522 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e659e40f-8ebe-4ef9-9c50-d5c1e0d69b8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:56.109030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e659e40f-8ebe-4ef9-9c50-d5c1e0d69b8b admin admin] PATH_INFO: `/v3/groups/2597fd1ab32f49e1a4978291427c998c` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:56.111023 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 130/264] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:55 2022] GET /identity/v3/roles/reader => generated 84 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:56.112661 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78736515-e118-4cd6-bc53-7c2b06ceed7c None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:56.121147 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e659e40f-8ebe-4ef9-9c50-d5c1e0d69b8b admin admin] RBAC: Authorizing `identity:get_group(group_id=2597fd1ab32f49e1a4978291427c998c)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:56.122712 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: WARNING py.warnings [None req-e659e40f-8ebe-4ef9-9c50-d5c1e0d69b8b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 16 00:43:56.122712 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: warnings.warn(msg) Oct 16 00:43:56.122712 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: Oct 16 00:43:56.124592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e659e40f-8ebe-4ef9-9c50-d5c1e0d69b8b admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:56.126687 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 135/265] 10.222.0.25 () {56 vars in 1201 bytes} [Sun Oct 16 00:43:56 2022] GET /identity/v3/groups/2597fd1ab32f49e1a4978291427c998c => generated 226 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:56.135633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e659e40f-8ebe-4ef9-9c50-d5c1e0d69b8b admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:56.153032 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e659e40f-8ebe-4ef9-9c50-d5c1e0d69b8b admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:56.156898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:56.163104 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fd9898-11ae-4180-86c0-b1cc333f7a06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:56.164244 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fd9898-11ae-4180-86c0-b1cc333f7a06 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:56.164759 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fd9898-11ae-4180-86c0-b1cc333f7a06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:56.165264 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fd9898-11ae-4180-86c0-b1cc333f7a06 admin admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:56.167105 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1fd9898-11ae-4180-86c0-b1cc333f7a06 admin admin] RBAC: Authorizing `identity:get_project(project_id=c3343320b0624293946f3195f2d767e2)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:56.169607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1fd9898-11ae-4180-86c0-b1cc333f7a06 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:56.172589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 136/266] 10.222.0.25 () {56 vars in 1207 bytes} [Sun Oct 16 00:43:56 2022] GET /identity/v3/projects/c3343320b0624293946f3195f2d767e2 => generated 298 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:56.180477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fd9898-11ae-4180-86c0-b1cc333f7a06 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:56.201884 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fd9898-11ae-4180-86c0-b1cc333f7a06 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:56.209998 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469a22a7-7570-472c-8240-c85aa9addb2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:56.211063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469a22a7-7570-472c-8240-c85aa9addb2f admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:56.211660 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469a22a7-7570-472c-8240-c85aa9addb2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:56.212170 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469a22a7-7570-472c-8240-c85aa9addb2f admin admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2/groups/2597fd1ab32f49e1a4978291427c998c/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:56.222595 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469a22a7-7570-472c-8240-c85aa9addb2f admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3343320b0624293946f3195f2d767e2, group_id=2597fd1ab32f49e1a4978291427c998c, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:56.224816 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: WARNING py.warnings [None req-469a22a7-7570-472c-8240-c85aa9addb2f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 16 00:43:56.224816 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: warnings.warn(msg) Oct 16 00:43:56.224816 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: Oct 16 00:43:56.227393 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469a22a7-7570-472c-8240-c85aa9addb2f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:56.256384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 137/267] 10.222.0.25 () {58 vars in 1463 bytes} [Sun Oct 16 00:43:56 2022] PUT /identity/v3/projects/c3343320b0624293946f3195f2d767e2/groups/2597fd1ab32f49e1a4978291427c998c/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:43:56.299627 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78736515-e118-4cd6-bc53-7c2b06ceed7c None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:56.320463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469a22a7-7570-472c-8240-c85aa9addb2f admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:56.366590 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5ee71e-acc0-43e7-a565-cc013b38f3f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:56.367246 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5ee71e-acc0-43e7-a565-cc013b38f3f5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:56.367535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5ee71e-acc0-43e7-a565-cc013b38f3f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:56.367810 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5ee71e-acc0-43e7-a565-cc013b38f3f5 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:56.368520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5ee71e-acc0-43e7-a565-cc013b38f3f5 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:56.369731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5ee71e-acc0-43e7-a565-cc013b38f3f5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:56.370461 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5ee71e-acc0-43e7-a565-cc013b38f3f5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:56.387268 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 131/268] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:43:56 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:56.393677 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5ee71e-acc0-43e7-a565-cc013b38f3f5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:56.444457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469a22a7-7570-472c-8240-c85aa9addb2f admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:56.450170 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2208a401-907d-41e1-97a3-37a4d7e6650d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:56.451831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2208a401-907d-41e1-97a3-37a4d7e6650d admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:56.452116 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2208a401-907d-41e1-97a3-37a4d7e6650d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:56.452391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2208a401-907d-41e1-97a3-37a4d7e6650d admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:56.471022 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2208a401-907d-41e1-97a3-37a4d7e6650d admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:56.476414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-2208a401-907d-41e1-97a3-37a4d7e6650d admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:43:56.476414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:56.476414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:56.476414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:56.476414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:56.476414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:56.476414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:56.476414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:56.476414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:56.476414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:56.476414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:56.476414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:56.476414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:56.476414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:56.476414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:56.476414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:56.477457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:56.477457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:56.477457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:56.477457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:56.477457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:56.477457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:56.477457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:56.477457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:56.477457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:56.477457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:56.477457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:56.477457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:56.477457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:56.477457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:56.477457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:56.477457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:56.478255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:56.478255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:56.478255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:56.478255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:56.478255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:56.478255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:56.478255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:56.478255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:56.478255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:56.478255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:56.478255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:43:56.478255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:56.478255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 138/269] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Oct 16 00:43:56 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:56.480440 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f14edc2-ee6d-4d33-bf12-79073bc84e32 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:56.480440 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f14edc2-ee6d-4d33-bf12-79073bc84e32 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:56.480740 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f14edc2-ee6d-4d33-bf12-79073bc84e32 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:56.482168 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 139/270] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:56.488496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f14edc2-ee6d-4d33-bf12-79073bc84e32 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:56.504639 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f14edc2-ee6d-4d33-bf12-79073bc84e32 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:56.510400 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecc56d3-f4bf-4f72-93bc-bcbe0cef08a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:56.511264 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecc56d3-f4bf-4f72-93bc-bcbe0cef08a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:56.511550 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecc56d3-f4bf-4f72-93bc-bcbe0cef08a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:56.511866 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecc56d3-f4bf-4f72-93bc-bcbe0cef08a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:56.512557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecc56d3-f4bf-4f72-93bc-bcbe0cef08a5 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:56.512856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecc56d3-f4bf-4f72-93bc-bcbe0cef08a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:56.515018 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecc56d3-f4bf-4f72-93bc-bcbe0cef08a5 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:56.526309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 140/271] 10.222.0.25 () {56 vars in 1132 bytes} [Sun Oct 16 00:43:56 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:56.527806 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9f2d3e-5f81-4ba4-b319-a4a25df2d572 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:56.528203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9f2d3e-5f81-4ba4-b319-a4a25df2d572 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:56.528480 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9f2d3e-5f81-4ba4-b319-a4a25df2d572 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:56.552758 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-4a9f2d3e-5f81-4ba4-b319-a4a25df2d572 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:56.554024 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4a9f2d3e-5f81-4ba4-b319-a4a25df2d572 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:56.564694 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 141/272] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:56 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 16 00:43:56.565581 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9f2d3e-5f81-4ba4-b319-a4a25df2d572 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:56.581773 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9f2d3e-5f81-4ba4-b319-a4a25df2d572 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:56.582300 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:56.600155 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df361c11-161c-453a-a7e7-adead34aa7d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:56.600820 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df361c11-161c-453a-a7e7-adead34aa7d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:56.601103 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df361c11-161c-453a-a7e7-adead34aa7d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:56.601381 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df361c11-161c-453a-a7e7-adead34aa7d4 admin admin] PATH_INFO: `/v3/groups/2597fd1ab32f49e1a4978291427c998c` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:56.604241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df361c11-161c-453a-a7e7-adead34aa7d4 admin admin] RBAC: Authorizing `identity:get_group(group_id=2597fd1ab32f49e1a4978291427c998c)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:56.608576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df361c11-161c-453a-a7e7-adead34aa7d4 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:56.620648 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 142/273] 10.222.0.25 () {56 vars in 1201 bytes} [Sun Oct 16 00:43:56 2022] GET /identity/v3/groups/2597fd1ab32f49e1a4978291427c998c => generated 226 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:56.627780 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df361c11-161c-453a-a7e7-adead34aa7d4 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:56.644809 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5ee71e-acc0-43e7-a565-cc013b38f3f5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:56.659263 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df361c11-161c-453a-a7e7-adead34aa7d4 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:56.679216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba19afe-b31d-4f58-a158-e1fbad8edca6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:56.679869 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba19afe-b31d-4f58-a158-e1fbad8edca6 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:56.680152 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba19afe-b31d-4f58-a158-e1fbad8edca6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:56.680430 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba19afe-b31d-4f58-a158-e1fbad8edca6 admin admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:56.682172 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba19afe-b31d-4f58-a158-e1fbad8edca6 admin admin] RBAC: Authorizing `identity:get_project(project_id=c3343320b0624293946f3195f2d767e2)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:56.686897 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba19afe-b31d-4f58-a158-e1fbad8edca6 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:56.686897 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 143/274] 10.222.0.25 () {56 vars in 1207 bytes} [Sun Oct 16 00:43:56 2022] GET /identity/v3/projects/c3343320b0624293946f3195f2d767e2 => generated 298 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:56.695610 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba19afe-b31d-4f58-a158-e1fbad8edca6 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:56.710183 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713c6731-d3d6-4f6f-91a4-afd2913e9c1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:56.712099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713c6731-d3d6-4f6f-91a4-afd2913e9c1b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:56.712099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713c6731-d3d6-4f6f-91a4-afd2913e9c1b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:56.712099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713c6731-d3d6-4f6f-91a4-afd2913e9c1b None admin] PATH_INFO: `/v3/users/40fd3e0b8a9a4c8eb5b9e2167be54acf` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:56.713515 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713c6731-d3d6-4f6f-91a4-afd2913e9c1b None admin] RBAC: Authorizing `identity:get_user(user_id=40fd3e0b8a9a4c8eb5b9e2167be54acf)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:56.715297 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713c6731-d3d6-4f6f-91a4-afd2913e9c1b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:56.716111 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba19afe-b31d-4f58-a158-e1fbad8edca6 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:56.722168 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 132/275] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:43:56 2022] GET /identity/v3/users/40fd3e0b8a9a4c8eb5b9e2167be54acf => generated 293 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:56.723755 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713c6731-d3d6-4f6f-91a4-afd2913e9c1b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:56.729511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d813b69b-5e0d-4bfa-b63c-02ff16fe175f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:56.730431 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d813b69b-5e0d-4bfa-b63c-02ff16fe175f admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:56.730854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d813b69b-5e0d-4bfa-b63c-02ff16fe175f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:56.731308 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d813b69b-5e0d-4bfa-b63c-02ff16fe175f admin admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2/groups/2597fd1ab32f49e1a4978291427c998c/roles/d9dcb4cfc0a245a98a7d96c100a497a1` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:56.739370 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d813b69b-5e0d-4bfa-b63c-02ff16fe175f admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3343320b0624293946f3195f2d767e2, group_id=2597fd1ab32f49e1a4978291427c998c, role_id=d9dcb4cfc0a245a98a7d96c100a497a1)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:56.742668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d813b69b-5e0d-4bfa-b63c-02ff16fe175f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:56.761194 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:56.768972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 144/276] 10.222.0.25 () {58 vars in 1463 bytes} [Sun Oct 16 00:43:56 2022] PUT /identity/v3/projects/c3343320b0624293946f3195f2d767e2/groups/2597fd1ab32f49e1a4978291427c998c/roles/d9dcb4cfc0a245a98a7d96c100a497a1 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:43:56.879938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713c6731-d3d6-4f6f-91a4-afd2913e9c1b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:56.926435 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8228809f-6dc5-46a1-bfbd-cb64605a7587 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:56.927128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8228809f-6dc5-46a1-bfbd-cb64605a7587 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:56.927396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8228809f-6dc5-46a1-bfbd-cb64605a7587 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:56.927674 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8228809f-6dc5-46a1-bfbd-cb64605a7587 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:56.928437 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8228809f-6dc5-46a1-bfbd-cb64605a7587 None admin] RBAC: Adding query filter params (role.id=1ac2f92c6eb34313ac9d9dfd4476663c, user.id=40fd3e0b8a9a4c8eb5b9e2167be54acf, scope.system=all) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:56.928752 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8228809f-6dc5-46a1-bfbd-cb64605a7587 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:56.930266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8228809f-6dc5-46a1-bfbd-cb64605a7587 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:56.944137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 133/277] 10.222.0.25 () {56 vars in 1308 bytes} [Sun Oct 16 00:43:56 2022] GET /identity/v3/role_assignments?user.id=40fd3e0b8a9a4c8eb5b9e2167be54acf&scope.system=all&role.id=1ac2f92c6eb34313ac9d9dfd4476663c => generated 510 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:57.099841 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d813b69b-5e0d-4bfa-b63c-02ff16fe175f admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:57.163604 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d397657c-29a0-4386-9a47-c68357788661 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:57.164094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d397657c-29a0-4386-9a47-c68357788661 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:57.164094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d397657c-29a0-4386-9a47-c68357788661 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:57.166989 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 134/278] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:57.174408 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729b63ff-b70a-4383-a10a-5f12978a8831 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:57.174757 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729b63ff-b70a-4383-a10a-5f12978a8831 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:57.175020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729b63ff-b70a-4383-a10a-5f12978a8831 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:57.188976 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:57.200810 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d813b69b-5e0d-4bfa-b63c-02ff16fe175f admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:57.211410 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add1ce75-bf77-4f44-a759-cbbb14e95173 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:57.212123 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add1ce75-bf77-4f44-a759-cbbb14e95173 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:57.212415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add1ce75-bf77-4f44-a759-cbbb14e95173 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:57.212693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add1ce75-bf77-4f44-a759-cbbb14e95173 admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:57.214751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-729b63ff-b70a-4383-a10a-5f12978a8831 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:57.217651 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-729b63ff-b70a-4383-a10a-5f12978a8831 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:57.236602 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 135/279] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:57 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 16 00:43:57.237165 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add1ce75-bf77-4f44-a759-cbbb14e95173 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:57.242797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-add1ce75-bf77-4f44-a759-cbbb14e95173 admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:57.242797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:57.242797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:57.242797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:57.242797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:57.242797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:57.242797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:57.242797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:57.242797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:57.242797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:57.242797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:57.242797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:57.242797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:57.242797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:57.242797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:57.242797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:57.243807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:57.243807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:57.243807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:57.243807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:57.243807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:57.243807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:57.243807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:57.243807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:57.243807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:57.243807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:57.243807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:57.243807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:57.243807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:57.243807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:57.243807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:57.243807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:57.244870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:57.244870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:57.244870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:57.244870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:57.244870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:57.244870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:57.244870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:57.244870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:57.244870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:57.244870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:57.244870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:43:57.244870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:57.244870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 145/280] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Oct 16 00:43:57 2022] GET /identity/v3/roles/member => generated 84 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:57.252125 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729b63ff-b70a-4383-a10a-5f12978a8831 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:57.264452 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729b63ff-b70a-4383-a10a-5f12978a8831 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:57.271952 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bccf371-a0ad-405b-a8da-2fedebdce705 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:57.272694 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bccf371-a0ad-405b-a8da-2fedebdce705 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:57.272984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bccf371-a0ad-405b-a8da-2fedebdce705 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:57.273315 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bccf371-a0ad-405b-a8da-2fedebdce705 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:57.273945 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bccf371-a0ad-405b-a8da-2fedebdce705 admin admin] RBAC: Adding query filter params (name=member) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:57.274326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bccf371-a0ad-405b-a8da-2fedebdce705 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:57.276506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: WARNING py.warnings [None req-7bccf371-a0ad-405b-a8da-2fedebdce705 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 16 00:43:57.276506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: warnings.warn(msg) Oct 16 00:43:57.276506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: Oct 16 00:43:57.277499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bccf371-a0ad-405b-a8da-2fedebdce705 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:57.292465 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 136/281] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 16 00:43:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:57.298432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add1ce75-bf77-4f44-a759-cbbb14e95173 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:57.311832 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add1ce75-bf77-4f44-a759-cbbb14e95173 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:57.319168 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097ea5ef-22eb-4b23-bb2d-7b1d91dd2bef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:57.319890 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097ea5ef-22eb-4b23-bb2d-7b1d91dd2bef admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:57.320176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097ea5ef-22eb-4b23-bb2d-7b1d91dd2bef admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:57.320504 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097ea5ef-22eb-4b23-bb2d-7b1d91dd2bef admin admin] PATH_INFO: `/v3/groups/2597fd1ab32f49e1a4978291427c998c` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:57.321935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097ea5ef-22eb-4b23-bb2d-7b1d91dd2bef admin admin] RBAC: Authorizing `identity:get_group(group_id=2597fd1ab32f49e1a4978291427c998c)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:57.324217 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097ea5ef-22eb-4b23-bb2d-7b1d91dd2bef admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:57.326482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 146/282] 10.222.0.25 () {56 vars in 1201 bytes} [Sun Oct 16 00:43:57 2022] GET /identity/v3/groups/2597fd1ab32f49e1a4978291427c998c => generated 226 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:57.332452 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bccf371-a0ad-405b-a8da-2fedebdce705 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:57.344227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bccf371-a0ad-405b-a8da-2fedebdce705 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:57.349948 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ffc3fa-7b8c-4188-8726-2e062dd1f797 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:57.350609 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ffc3fa-7b8c-4188-8726-2e062dd1f797 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:57.350956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ffc3fa-7b8c-4188-8726-2e062dd1f797 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:57.351197 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ffc3fa-7b8c-4188-8726-2e062dd1f797 admin admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:57.355773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ffc3fa-7b8c-4188-8726-2e062dd1f797 admin admin] RBAC: Authorizing `identity:get_project(project_id=06615629856547fd9da00a208187c681)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:57.357952 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ffc3fa-7b8c-4188-8726-2e062dd1f797 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:57.360115 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 137/283] 10.222.0.25 () {56 vars in 1207 bytes} [Sun Oct 16 00:43:57 2022] GET /identity/v3/projects/06615629856547fd9da00a208187c681 => generated 302 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:57.366018 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097ea5ef-22eb-4b23-bb2d-7b1d91dd2bef admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:57.380934 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097ea5ef-22eb-4b23-bb2d-7b1d91dd2bef admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:57.388723 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009ccd8b-024d-42b6-b997-0cef2e1c3631 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:57.389795 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009ccd8b-024d-42b6-b997-0cef2e1c3631 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:57.390368 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009ccd8b-024d-42b6-b997-0cef2e1c3631 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:57.390908 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009ccd8b-024d-42b6-b997-0cef2e1c3631 admin admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681/groups/2597fd1ab32f49e1a4978291427c998c/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:57.394178 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009ccd8b-024d-42b6-b997-0cef2e1c3631 admin admin] RBAC: Authorizing `identity:create_grant(project_id=06615629856547fd9da00a208187c681, group_id=2597fd1ab32f49e1a4978291427c998c, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:57.397419 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009ccd8b-024d-42b6-b997-0cef2e1c3631 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:57.418559 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 147/284] 10.222.0.25 () {58 vars in 1463 bytes} [Sun Oct 16 00:43:57 2022] PUT /identity/v3/projects/06615629856547fd9da00a208187c681/groups/2597fd1ab32f49e1a4978291427c998c/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:43:57.832950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ffc3fa-7b8c-4188-8726-2e062dd1f797 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:57.844041 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ffc3fa-7b8c-4188-8726-2e062dd1f797 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:57.924154 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78eab218-0b07-4abd-80c5-4441e54ca3ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:57.925069 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78eab218-0b07-4abd-80c5-4441e54ca3ba admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:57.925536 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78eab218-0b07-4abd-80c5-4441e54ca3ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:57.925962 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78eab218-0b07-4abd-80c5-4441e54ca3ba admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:57.944019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78eab218-0b07-4abd-80c5-4441e54ca3ba admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:57.950615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-78eab218-0b07-4abd-80c5-4441e54ca3ba admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:43:57.950615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:57.950615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:57.950615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:57.950615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:57.950615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:57.950615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:57.950615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:57.950615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:57.950615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:57.950615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:57.950615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:57.950615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:57.950615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:57.950615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:57.950615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:57.951789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:57.951789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:57.951789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:57.951789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:57.951789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:57.951789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:57.951789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:57.951789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:57.951789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:57.951789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:57.951789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:57.951789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:57.951789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:57.951789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:57.951789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:57.951789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:57.952795 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:57.952795 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:57.952795 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:57.952795 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:57.952795 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:57.952795 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:57.952795 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:57.952795 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:57.952795 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:57.952795 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:57.952795 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:43:57.952795 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:43:57.954880 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 138/285] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Oct 16 00:43:57 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 122 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:43:57.961100 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009ccd8b-024d-42b6-b997-0cef2e1c3631 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:57.974018 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009ccd8b-024d-42b6-b997-0cef2e1c3631 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:57.979553 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08a7413-75df-4eda-8626-1b57ace5272a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:57.980498 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08a7413-75df-4eda-8626-1b57ace5272a admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:57.980978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08a7413-75df-4eda-8626-1b57ace5272a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:57.981403 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08a7413-75df-4eda-8626-1b57ace5272a admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:57.982299 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08a7413-75df-4eda-8626-1b57ace5272a admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:57.982766 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08a7413-75df-4eda-8626-1b57ace5272a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:57.984799 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08a7413-75df-4eda-8626-1b57ace5272a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:58.004944 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 148/286] 10.222.0.25 () {56 vars in 1132 bytes} [Sun Oct 16 00:43:57 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:58.012647 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78eab218-0b07-4abd-80c5-4441e54ca3ba admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:58.024375 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78eab218-0b07-4abd-80c5-4441e54ca3ba admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:58.030109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc74f71d-adda-4512-bcf7-d81b9518af1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:58.031164 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc74f71d-adda-4512-bcf7-d81b9518af1f admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:58.031604 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc74f71d-adda-4512-bcf7-d81b9518af1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:58.032079 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc74f71d-adda-4512-bcf7-d81b9518af1f admin admin] PATH_INFO: `/v3/groups/2597fd1ab32f49e1a4978291427c998c` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:58.033470 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc74f71d-adda-4512-bcf7-d81b9518af1f admin admin] RBAC: Authorizing `identity:get_group(group_id=2597fd1ab32f49e1a4978291427c998c)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:58.034731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: WARNING py.warnings [None req-fc74f71d-adda-4512-bcf7-d81b9518af1f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 16 00:43:58.034731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: warnings.warn(msg) Oct 16 00:43:58.034731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: Oct 16 00:43:58.036386 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc74f71d-adda-4512-bcf7-d81b9518af1f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:58.038380 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 139/287] 10.222.0.25 () {56 vars in 1201 bytes} [Sun Oct 16 00:43:58 2022] GET /identity/v3/groups/2597fd1ab32f49e1a4978291427c998c => generated 226 bytes in 26 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:58.059147 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08a7413-75df-4eda-8626-1b57ace5272a admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:58.079643 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08a7413-75df-4eda-8626-1b57ace5272a admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:58.087918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb42d50-a521-456e-ab3b-4b7ad958e893 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:58.089160 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb42d50-a521-456e-ab3b-4b7ad958e893 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:58.089613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb42d50-a521-456e-ab3b-4b7ad958e893 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:58.090227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb42d50-a521-456e-ab3b-4b7ad958e893 admin admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:58.092107 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb42d50-a521-456e-ab3b-4b7ad958e893 admin admin] RBAC: Authorizing `identity:get_project(project_id=06615629856547fd9da00a208187c681)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:58.095386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb42d50-a521-456e-ab3b-4b7ad958e893 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:58.099252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 149/288] 10.222.0.25 () {56 vars in 1207 bytes} [Sun Oct 16 00:43:58 2022] GET /identity/v3/projects/06615629856547fd9da00a208187c681 => generated 302 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:58.105335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc74f71d-adda-4512-bcf7-d81b9518af1f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:58.121403 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc74f71d-adda-4512-bcf7-d81b9518af1f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:58.136394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a5e87e-89da-41df-bd95-beb32b51a64c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:58.138341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a5e87e-89da-41df-bd95-beb32b51a64c admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:58.139124 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a5e87e-89da-41df-bd95-beb32b51a64c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:58.140026 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a5e87e-89da-41df-bd95-beb32b51a64c admin admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681/groups/2597fd1ab32f49e1a4978291427c998c/roles/d9dcb4cfc0a245a98a7d96c100a497a1` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:58.152714 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a5e87e-89da-41df-bd95-beb32b51a64c admin admin] RBAC: Authorizing `identity:create_grant(project_id=06615629856547fd9da00a208187c681, group_id=2597fd1ab32f49e1a4978291427c998c, role_id=d9dcb4cfc0a245a98a7d96c100a497a1)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:58.155107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: WARNING py.warnings [None req-58a5e87e-89da-41df-bd95-beb32b51a64c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 16 00:43:58.155107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: warnings.warn(msg) Oct 16 00:43:58.155107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: Oct 16 00:43:58.159585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a5e87e-89da-41df-bd95-beb32b51a64c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:58.184726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 140/289] 10.222.0.25 () {58 vars in 1463 bytes} [Sun Oct 16 00:43:58 2022] PUT /identity/v3/projects/06615629856547fd9da00a208187c681/groups/2597fd1ab32f49e1a4978291427c998c/roles/d9dcb4cfc0a245a98a7d96c100a497a1 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:43:58.393256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02f65d6-c55f-422a-b37e-2e0001b1b5df None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:58.394162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02f65d6-c55f-422a-b37e-2e0001b1b5df None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:58.394162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02f65d6-c55f-422a-b37e-2e0001b1b5df None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:58.395150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 150/290] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:58.409484 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5c67fc-5729-42d2-865a-a5d36f0d9fc7 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:58.410299 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5c67fc-5729-42d2-865a-a5d36f0d9fc7 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:58.411129 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5c67fc-5729-42d2-865a-a5d36f0d9fc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:58.450087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5b5c67fc-5729-42d2-865a-a5d36f0d9fc7 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:58.452677 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5b5c67fc-5729-42d2-865a-a5d36f0d9fc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:58.545523 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 141/291] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:58 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 16 00:43:59.106493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02f65d6-c55f-422a-b37e-2e0001b1b5df None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:59.120901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02f65d6-c55f-422a-b37e-2e0001b1b5df None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:59.130658 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ef795b-3d2c-4a75-941e-778151cf61fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:59.131774 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ef795b-3d2c-4a75-941e-778151cf61fa admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:59.132351 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ef795b-3d2c-4a75-941e-778151cf61fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:59.132854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ef795b-3d2c-4a75-941e-778151cf61fa admin admin] PATH_INFO: `/v3/roles/admin` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:59.150130 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ef795b-3d2c-4a75-941e-778151cf61fa admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:43:59.157096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-d4ef795b-3d2c-4a75-941e-778151cf61fa admin admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:43:59.157096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:43:59.157096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:43:59.157096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:43:59.157096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:43:59.157096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:43:59.157096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:43:59.157096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:43:59.157096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:43:59.157096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:43:59.157096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:43:59.157096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:43:59.157096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:43:59.157096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:43:59.157096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:43:59.157096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:43:59.158159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:43:59.158159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:43:59.158159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:43:59.158159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:43:59.158159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:43:59.158159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:43:59.158159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:43:59.158159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:43:59.158159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:43:59.158159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:43:59.158159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:43:59.158159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:43:59.158159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:43:59.158159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:43:59.158159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:43:59.158159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:43:59.159069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:43:59.159069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:43:59.159069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:43:59.159069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:43:59.159069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:43:59.159069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:43:59.159069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:43:59.159069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:43:59.159069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:43:59.159069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:43:59.159069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:43:59.159069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:43:59.161646 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 151/292] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 16 00:43:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 56 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:43:59.171127 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5c67fc-5729-42d2-865a-a5d36f0d9fc7 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:59.182803 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5c67fc-5729-42d2-865a-a5d36f0d9fc7 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:59.189798 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d21f5d-2d19-4b25-90ba-783847758afb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:59.191382 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d21f5d-2d19-4b25-90ba-783847758afb admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:59.191889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d21f5d-2d19-4b25-90ba-783847758afb admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:59.192396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d21f5d-2d19-4b25-90ba-783847758afb admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:59.193345 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d21f5d-2d19-4b25-90ba-783847758afb admin admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:43:59.193892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d21f5d-2d19-4b25-90ba-783847758afb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:59.196093 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d21f5d-2d19-4b25-90ba-783847758afb admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:59.210984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 142/293] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Oct 16 00:43:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:59.219980 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ef795b-3d2c-4a75-941e-778151cf61fa admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:59.239684 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ef795b-3d2c-4a75-941e-778151cf61fa admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:59.249708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c57c9f-0a26-4217-a174-8a9713e58d2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:59.255425 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c57c9f-0a26-4217-a174-8a9713e58d2a admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:59.256095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c57c9f-0a26-4217-a174-8a9713e58d2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:59.256566 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c57c9f-0a26-4217-a174-8a9713e58d2a admin admin] PATH_INFO: `/v3/groups/b7e1cc2052ba47cc9e29d0ad9c80d4bf` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:59.273200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c57c9f-0a26-4217-a174-8a9713e58d2a admin admin] RBAC: Authorizing `identity:get_group(group_id=b7e1cc2052ba47cc9e29d0ad9c80d4bf)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:59.277403 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c57c9f-0a26-4217-a174-8a9713e58d2a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:59.280999 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 152/294] 10.222.0.25 () {56 vars in 1201 bytes} [Sun Oct 16 00:43:59 2022] GET /identity/v3/groups/b7e1cc2052ba47cc9e29d0ad9c80d4bf => generated 229 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:43:59.289847 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d21f5d-2d19-4b25-90ba-783847758afb admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:59.305764 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d21f5d-2d19-4b25-90ba-783847758afb admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:59.312582 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46bb583-f3fa-4540-b2cb-5ee406718e8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:59.313699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46bb583-f3fa-4540-b2cb-5ee406718e8d admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:59.314202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46bb583-f3fa-4540-b2cb-5ee406718e8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:59.314734 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46bb583-f3fa-4540-b2cb-5ee406718e8d admin admin] PATH_INFO: `/v3/projects/f1d89d5c90cd4c35b26fbcfc82450844` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:59.319910 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46bb583-f3fa-4540-b2cb-5ee406718e8d admin admin] RBAC: Authorizing `identity:get_project(project_id=f1d89d5c90cd4c35b26fbcfc82450844)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:59.322905 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46bb583-f3fa-4540-b2cb-5ee406718e8d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:59.325468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 143/295] 10.222.0.25 () {56 vars in 1207 bytes} [Sun Oct 16 00:43:59 2022] GET /identity/v3/projects/f1d89d5c90cd4c35b26fbcfc82450844 => generated 344 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:43:59.335367 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c57c9f-0a26-4217-a174-8a9713e58d2a admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:43:59.360740 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c57c9f-0a26-4217-a174-8a9713e58d2a admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:43:59.372322 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4bf108-0015-4c2b-ad3a-9dbf5095ceaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:43:59.373357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4bf108-0015-4c2b-ad3a-9dbf5095ceaa admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:59.373833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4bf108-0015-4c2b-ad3a-9dbf5095ceaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:59.374311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4bf108-0015-4c2b-ad3a-9dbf5095ceaa admin admin] PATH_INFO: `/v3/projects/f1d89d5c90cd4c35b26fbcfc82450844/groups/b7e1cc2052ba47cc9e29d0ad9c80d4bf/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:59.377876 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4bf108-0015-4c2b-ad3a-9dbf5095ceaa admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1d89d5c90cd4c35b26fbcfc82450844, group_id=b7e1cc2052ba47cc9e29d0ad9c80d4bf, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:43:59.380731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661eb358-e8af-45f5-9347-4b7d7d68a729 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:59.381304 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661eb358-e8af-45f5-9347-4b7d7d68a729 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:59.381784 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661eb358-e8af-45f5-9347-4b7d7d68a729 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:59.383353 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4bf108-0015-4c2b-ad3a-9dbf5095ceaa admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:43:59.383957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 144/296] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:43:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:43:59.391100 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3505c9-2e01-45c2-b03c-18358f95ec36 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:43:59.391730 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3505c9-2e01-45c2-b03c-18358f95ec36 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:43:59.392279 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3505c9-2e01-45c2-b03c-18358f95ec36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:43:59.417876 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 153/297] 10.222.0.25 () {58 vars in 1463 bytes} [Sun Oct 16 00:43:59 2022] PUT /identity/v3/projects/f1d89d5c90cd4c35b26fbcfc82450844/groups/b7e1cc2052ba47cc9e29d0ad9c80d4bf/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:43:59.433344 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ea3505c9-2e01-45c2-b03c-18358f95ec36 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:43:59.434954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ea3505c9-2e01-45c2-b03c-18358f95ec36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:43:59.482326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:43:59.535463 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 145/298] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:43:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:00.000268 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4bf108-0015-4c2b-ad3a-9dbf5095ceaa admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:00.024783 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921f2b9f-5212-4912-86c3-0969f7e13067 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:00.025198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921f2b9f-5212-4912-86c3-0969f7e13067 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:00.025750 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921f2b9f-5212-4912-86c3-0969f7e13067 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:00.028635 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 146/299] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:00 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:00.037774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9ee868-27c0-463b-8b78-3f2ad8552959 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:00.037774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9ee868-27c0-463b-8b78-3f2ad8552959 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:00.037774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9ee868-27c0-463b-8b78-3f2ad8552959 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:00.096615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-8f9ee868-27c0-463b-8b78-3f2ad8552959 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:00.098583 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8f9ee868-27c0-463b-8b78-3f2ad8552959 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:00.106491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:00.160833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4bf108-0015-4c2b-ad3a-9dbf5095ceaa admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:00.281146 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37f80d9-837c-445c-b518-3e38d8ed2201 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:00.281731 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37f80d9-837c-445c-b518-3e38d8ed2201 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:00.282019 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37f80d9-837c-445c-b518-3e38d8ed2201 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:00.282317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37f80d9-837c-445c-b518-3e38d8ed2201 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:00.300323 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 147/300] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:00.311609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37f80d9-837c-445c-b518-3e38d8ed2201 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:00.317139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-b37f80d9-837c-445c-b518-3e38d8ed2201 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:44:00.317139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:00.317139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:00.317139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:00.317139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:00.317139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:00.317139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:00.317139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:00.317139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:00.317139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:00.317139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:00.317139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:00.317139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:00.317139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:00.317139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:00.317139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:00.318071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:00.318071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:00.318071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:00.318071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:00.318071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:00.318071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:00.318071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:00.318071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:00.318071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:00.318071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:00.318071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:00.318071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:00.318071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:00.318071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:00.318071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:00.318071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:00.318920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:00.318920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:00.318920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:00.318920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:00.318920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:00.318920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:00.318920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:00.318920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:00.318920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:00.318920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:00.318920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:44:00.318920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:00.318920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 154/301] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:43:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:00.324348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9ee868-27c0-463b-8b78-3f2ad8552959 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:00.463714 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:00.468661 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9ee868-27c0-463b-8b78-3f2ad8552959 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:00.516371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e5f5fc-39af-4758-bcc7-56792dba7ce3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:00.517009 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e5f5fc-39af-4758-bcc7-56792dba7ce3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:00.517295 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e5f5fc-39af-4758-bcc7-56792dba7ce3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:00.517576 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e5f5fc-39af-4758-bcc7-56792dba7ce3 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:00.518278 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e5f5fc-39af-4758-bcc7-56792dba7ce3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:00.518582 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e5f5fc-39af-4758-bcc7-56792dba7ce3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:00.520131 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e5f5fc-39af-4758-bcc7-56792dba7ce3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:00.532530 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 148/302] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:44:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:00.538667 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37f80d9-837c-445c-b518-3e38d8ed2201 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:00.665549 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37f80d9-837c-445c-b518-3e38d8ed2201 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:00.707918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f49b675-b3dc-44aa-b125-b80734c9ca32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:00.708574 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f49b675-b3dc-44aa-b125-b80734c9ca32 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:00.708859 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f49b675-b3dc-44aa-b125-b80734c9ca32 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:00.709141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f49b675-b3dc-44aa-b125-b80734c9ca32 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:00.713677 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:00.782978 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e5f5fc-39af-4758-bcc7-56792dba7ce3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:00.797524 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f49b675-b3dc-44aa-b125-b80734c9ca32 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:00.799078 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f49b675-b3dc-44aa-b125-b80734c9ca32 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:00.939970 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 155/303] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:44:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:44:00.960883 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f49b675-b3dc-44aa-b125-b80734c9ca32 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:00.986295 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e5f5fc-39af-4758-bcc7-56792dba7ce3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:01.033169 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4b8d23-de08-482d-b3e9-30df246a2dfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:01.034123 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4b8d23-de08-482d-b3e9-30df246a2dfc None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:01.034604 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4b8d23-de08-482d-b3e9-30df246a2dfc None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:01.035090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4b8d23-de08-482d-b3e9-30df246a2dfc None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:01.039530 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:01.061846 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4b8d23-de08-482d-b3e9-30df246a2dfc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:01.068851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-9b4b8d23-de08-482d-b3e9-30df246a2dfc None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:44:01.068851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:01.068851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:01.068851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:01.068851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:01.068851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:01.068851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:01.068851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:01.068851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:01.068851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:01.068851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:01.068851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:01.068851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:01.068851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:01.068851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:01.068851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:01.069829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:01.069829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:01.069829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:01.069829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:01.069829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:01.069829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:01.069829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:01.069829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:01.069829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:01.069829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:01.069829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:01.069829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:01.069829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:01.069829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:01.069829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:01.069829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:01.070634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:01.070634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:01.070634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:01.070634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:01.070634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:01.070634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:01.070634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:01.070634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:01.070634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:01.070634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:01.070634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:44:01.070634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:01.072321 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 149/304] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 16 00:44:00 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:01.079052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4b8d23-de08-482d-b3e9-30df246a2dfc None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:01.130206 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:01.188322 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f49b675-b3dc-44aa-b125-b80734c9ca32 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:01.239857 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b621991d-bb63-4975-8486-404fc7211d7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:01.241955 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b621991d-bb63-4975-8486-404fc7211d7c None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:01.242636 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b621991d-bb63-4975-8486-404fc7211d7c None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:01.245315 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b621991d-bb63-4975-8486-404fc7211d7c None admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:01.247692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b621991d-bb63-4975-8486-404fc7211d7c None admin] RBAC: Authorizing `identity:get_project(project_id=c3343320b0624293946f3195f2d767e2)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:01.249419 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b621991d-bb63-4975-8486-404fc7211d7c None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:01.256114 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 156/305] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:44:00 2022] GET /identity/v3/projects/c3343320b0624293946f3195f2d767e2 => generated 298 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:01.263426 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b621991d-bb63-4975-8486-404fc7211d7c None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:01.328735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4b8d23-de08-482d-b3e9-30df246a2dfc None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:01.389246 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70651e2-aece-4e1d-8612-2c12c82ff541 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:01.389873 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70651e2-aece-4e1d-8612-2c12c82ff541 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:01.390143 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70651e2-aece-4e1d-8612-2c12c82ff541 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:01.390462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70651e2-aece-4e1d-8612-2c12c82ff541 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:01.391254 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70651e2-aece-4e1d-8612-2c12c82ff541 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:01.391568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70651e2-aece-4e1d-8612-2c12c82ff541 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:01.393241 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70651e2-aece-4e1d-8612-2c12c82ff541 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:01.409691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 150/306] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 332 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:01.411124 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1b835d-883c-4ad2-9ab9-03b6c4f02a9c None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:01.411411 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1b835d-883c-4ad2-9ab9-03b6c4f02a9c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:01.411674 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1b835d-883c-4ad2-9ab9-03b6c4f02a9c None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:01.411835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b621991d-bb63-4975-8486-404fc7211d7c None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:01.413057 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 151/307] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:01.418337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1b835d-883c-4ad2-9ab9-03b6c4f02a9c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:01.420506 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:01.463635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6847100-afd0-4c2d-aab2-260fa3201986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:01.464389 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6847100-afd0-4c2d-aab2-260fa3201986 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:01.464773 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6847100-afd0-4c2d-aab2-260fa3201986 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:01.465148 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6847100-afd0-4c2d-aab2-260fa3201986 None admin] PATH_INFO: `/v3/groups/2597fd1ab32f49e1a4978291427c998c` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:01.466535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6847100-afd0-4c2d-aab2-260fa3201986 None admin] RBAC: Authorizing `identity:get_group(group_id=2597fd1ab32f49e1a4978291427c998c)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:01.468362 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6847100-afd0-4c2d-aab2-260fa3201986 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:01.472207 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 157/308] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 16 00:44:01 2022] GET /identity/v3/groups/2597fd1ab32f49e1a4978291427c998c => generated 226 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:01.473352 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256cdd0e-d54f-4b56-b154-acbf58317ed1 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:01.473638 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256cdd0e-d54f-4b56-b154-acbf58317ed1 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:01.473905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256cdd0e-d54f-4b56-b154-acbf58317ed1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:01.501901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-256cdd0e-d54f-4b56-b154-acbf58317ed1 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:01.503340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-256cdd0e-d54f-4b56-b154-acbf58317ed1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:01.543827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:01.583368 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1b835d-883c-4ad2-9ab9-03b6c4f02a9c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:01.607725 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 158/309] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:01.613028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256cdd0e-d54f-4b56-b154-acbf58317ed1 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:01.643808 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4b3cde-93ea-4060-95ca-d80933850f4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:01.644453 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4b3cde-93ea-4060-95ca-d80933850f4f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:01.644739 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4b3cde-93ea-4060-95ca-d80933850f4f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:01.647119 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4b3cde-93ea-4060-95ca-d80933850f4f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:01.776611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4b3cde-93ea-4060-95ca-d80933850f4f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:01.778439 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4b3cde-93ea-4060-95ca-d80933850f4f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:01.789182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256cdd0e-d54f-4b56-b154-acbf58317ed1 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:01.790961 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:01.861039 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e819f26d-c76e-4d33-85dc-6d6ba796a9bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:01.861961 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e819f26d-c76e-4d33-85dc-6d6ba796a9bc None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:01.862395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e819f26d-c76e-4d33-85dc-6d6ba796a9bc None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:01.862833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e819f26d-c76e-4d33-85dc-6d6ba796a9bc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:01.863758 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e819f26d-c76e-4d33-85dc-6d6ba796a9bc None admin] RBAC: Adding query filter params (group.id=2597fd1ab32f49e1a4978291427c998c, role.id=8a6c7a6c7d154f3db6d2639609d60224, scope.project.id=c3343320b0624293946f3195f2d767e2) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:01.864221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e819f26d-c76e-4d33-85dc-6d6ba796a9bc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:01.865845 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e819f26d-c76e-4d33-85dc-6d6ba796a9bc None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:01.878969 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 159/310] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Oct 16 00:44:01 2022] GET /identity/v3/role_assignments?group.id=2597fd1ab32f49e1a4978291427c998c&scope.project.id=c3343320b0624293946f3195f2d767e2&role.id=8a6c7a6c7d154f3db6d2639609d60224 => generated 611 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:01.886324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:01.905047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 152/311] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:44:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 487 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:44:01.911143 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e819f26d-c76e-4d33-85dc-6d6ba796a9bc None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:02.021696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4b3cde-93ea-4060-95ca-d80933850f4f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:02.076416 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e819f26d-c76e-4d33-85dc-6d6ba796a9bc None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:02.151827 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2f864f-978e-4b98-8616-112389c17e84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:02.152786 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2f864f-978e-4b98-8616-112389c17e84 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:02.153229 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2f864f-978e-4b98-8616-112389c17e84 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:02.153650 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2f864f-978e-4b98-8616-112389c17e84 None admin] PATH_INFO: `/v3/projects/c3343320b0624293946f3195f2d767e2` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:02.155226 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2f864f-978e-4b98-8616-112389c17e84 None admin] RBAC: Authorizing `identity:get_project(project_id=c3343320b0624293946f3195f2d767e2)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:02.156976 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2f864f-978e-4b98-8616-112389c17e84 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:02.162794 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 160/312] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:44:01 2022] GET /identity/v3/projects/c3343320b0624293946f3195f2d767e2 => generated 298 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:02.164128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4b3cde-93ea-4060-95ca-d80933850f4f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:02.172069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2f864f-978e-4b98-8616-112389c17e84 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:02.243695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02609793-100c-4f18-a611-618ab318e1ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:02.244594 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02609793-100c-4f18-a611-618ab318e1ad None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:02.244880 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02609793-100c-4f18-a611-618ab318e1ad None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:02.245156 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02609793-100c-4f18-a611-618ab318e1ad None admin] PATH_INFO: `/v3/roles/member` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:02.249974 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:02.270460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02609793-100c-4f18-a611-618ab318e1ad None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:02.275786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-02609793-100c-4f18-a611-618ab318e1ad None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:44:02.275786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:02.275786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:02.275786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:02.275786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:02.275786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:02.275786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:02.275786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:02.275786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:02.275786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:02.275786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:02.275786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:02.275786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:02.275786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:02.275786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:02.275786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:02.276950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:02.276950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:02.276950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:02.276950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:02.276950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:02.276950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:02.276950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:02.276950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:02.276950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:02.276950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:02.276950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:02.276950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:02.276950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:02.276950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:02.276950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:02.276950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:02.278192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:02.278192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:02.278192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:02.278192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:02.278192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:02.278192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:02.278192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:02.278192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:02.278192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:02.278192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:02.278192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 16 00:44:02.278192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:02.278192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 153/313] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:44:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:02.284155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02609793-100c-4f18-a611-618ab318e1ad None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:02.354767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:02.387224 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2f864f-978e-4b98-8616-112389c17e84 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:02.439037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02609793-100c-4f18-a611-618ab318e1ad None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:02.440663 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b46b49c-3dd8-464a-9cfe-683379121e9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:02.440806 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b46b49c-3dd8-464a-9cfe-683379121e9e None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:02.441295 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b46b49c-3dd8-464a-9cfe-683379121e9e None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:02.442888 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b46b49c-3dd8-464a-9cfe-683379121e9e None admin] PATH_INFO: `/v3/groups/2597fd1ab32f49e1a4978291427c998c` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:02.445099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b46b49c-3dd8-464a-9cfe-683379121e9e None admin] RBAC: Authorizing `identity:get_group(group_id=2597fd1ab32f49e1a4978291427c998c)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:02.448367 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b46b49c-3dd8-464a-9cfe-683379121e9e None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:02.451694 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 161/314] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 16 00:44:02 2022] GET /identity/v3/groups/2597fd1ab32f49e1a4978291427c998c => generated 226 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:02.453679 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6669cba1-025a-4e01-9660-af63a2b2a887 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:02.454211 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6669cba1-025a-4e01-9660-af63a2b2a887 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:02.454623 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6669cba1-025a-4e01-9660-af63a2b2a887 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:02.457511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 162/315] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:02.460081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815c8807-10b2-4bd1-9bfe-9dcb60ef829c None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:02.460753 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815c8807-10b2-4bd1-9bfe-9dcb60ef829c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:02.461110 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815c8807-10b2-4bd1-9bfe-9dcb60ef829c None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:02.462652 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 163/316] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:02.463980 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815c8807-10b2-4bd1-9bfe-9dcb60ef829c None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:02.482112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fe4076-4004-493a-a091-f10260ff7afe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:02.483190 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fe4076-4004-493a-a091-f10260ff7afe None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:02.483678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fe4076-4004-493a-a091-f10260ff7afe None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:02.484154 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fe4076-4004-493a-a091-f10260ff7afe None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:02.485334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fe4076-4004-493a-a091-f10260ff7afe None admin] RBAC: Adding query filter params (name=member) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:02.485863 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fe4076-4004-493a-a091-f10260ff7afe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:02.488509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fe4076-4004-493a-a091-f10260ff7afe None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:02.505363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 154/317] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:44:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:02.506979 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5c71dc-088d-4313-8ee8-8c8e2bed7781 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:02.507273 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5c71dc-088d-4313-8ee8-8c8e2bed7781 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:02.507551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5c71dc-088d-4313-8ee8-8c8e2bed7781 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:02.528253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-bf5c71dc-088d-4313-8ee8-8c8e2bed7781 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:02.530028 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bf5c71dc-088d-4313-8ee8-8c8e2bed7781 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:02.627770 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 155/318] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:02.630430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb075c9-9427-4078-8d2e-d12942001f88 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:02.631039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb075c9-9427-4078-8d2e-d12942001f88 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:02.631515 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb075c9-9427-4078-8d2e-d12942001f88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:02.632100 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815c8807-10b2-4bd1-9bfe-9dcb60ef829c None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:02.633841 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:02.667188 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5eb075c9-9427-4078-8d2e-d12942001f88 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:02.669111 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5eb075c9-9427-4078-8d2e-d12942001f88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:02.689017 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f685fa41-8c81-4b71-95f5-d149c7386973 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:02.689927 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f685fa41-8c81-4b71-95f5-d149c7386973 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:02.690379 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f685fa41-8c81-4b71-95f5-d149c7386973 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:02.691058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f685fa41-8c81-4b71-95f5-d149c7386973 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:02.692122 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f685fa41-8c81-4b71-95f5-d149c7386973 None admin] RBAC: Adding query filter params (group.id=2597fd1ab32f49e1a4978291427c998c, role.id=d9dcb4cfc0a245a98a7d96c100a497a1, scope.project.id=c3343320b0624293946f3195f2d767e2) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:02.692590 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f685fa41-8c81-4b71-95f5-d149c7386973 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:02.694530 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f685fa41-8c81-4b71-95f5-d149c7386973 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:02.704362 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 164/319] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Oct 16 00:44:02 2022] GET /identity/v3/role_assignments?group.id=2597fd1ab32f49e1a4978291427c998c&scope.project.id=c3343320b0624293946f3195f2d767e2&role.id=d9dcb4cfc0a245a98a7d96c100a497a1 => generated 611 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:02.705728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f685fa41-8c81-4b71-95f5-d149c7386973 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:02.728969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:02.847075 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 156/320] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:02.876083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f685fa41-8c81-4b71-95f5-d149c7386973 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:02.924753 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09fac65-5ead-4c16-8ce2-c437081b52cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:02.925824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09fac65-5ead-4c16-8ce2-c437081b52cd None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:02.926294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09fac65-5ead-4c16-8ce2-c437081b52cd None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:02.926760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09fac65-5ead-4c16-8ce2-c437081b52cd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:02.928639 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb075c9-9427-4078-8d2e-d12942001f88 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:02.985864 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:03.033199 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09fac65-5ead-4c16-8ce2-c437081b52cd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:03.034731 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09fac65-5ead-4c16-8ce2-c437081b52cd None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:03.097769 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb075c9-9427-4078-8d2e-d12942001f88 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:03.161453 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733b0d86-c4c3-49b5-9f7b-d178c9fe739a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:03.162122 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733b0d86-c4c3-49b5-9f7b-d178c9fe739a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:03.162408 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733b0d86-c4c3-49b5-9f7b-d178c9fe739a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:03.162683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733b0d86-c4c3-49b5-9f7b-d178c9fe739a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:03.177558 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733b0d86-c4c3-49b5-9f7b-d178c9fe739a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:03.180940 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 165/321] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:44:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 476 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:44:03.182711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-733b0d86-c4c3-49b5-9f7b-d178c9fe739a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:44:03.182711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:03.182711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:03.182711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:03.182711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:03.182711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:03.182711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:03.182711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:03.182711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:03.182711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:03.182711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:03.182711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:03.182711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:03.182711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:03.182711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:03.182711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:03.183717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:03.183717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:03.183717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:03.183717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:03.183717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:03.183717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:03.183717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:03.183717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:03.183717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:03.183717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:03.183717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:03.183717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:03.183717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:03.183717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:03.183717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:03.183717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:03.184526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:03.184526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:03.184526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:03.184526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:03.184526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:03.184526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:03.184526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:03.184526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:03.184526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:03.184526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:03.184526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:44:03.184526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:03.184526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 157/322] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:44:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:03.191438 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09fac65-5ead-4c16-8ce2-c437081b52cd None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:03.193951 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733b0d86-c4c3-49b5-9f7b-d178c9fe739a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:03.238474 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:03.287395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:03.333732 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733b0d86-c4c3-49b5-9f7b-d178c9fe739a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:03.354140 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09fac65-5ead-4c16-8ce2-c437081b52cd None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:03.380681 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acef9c65-9be5-4aa3-8c98-a590b6f87d1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:03.381354 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acef9c65-9be5-4aa3-8c98-a590b6f87d1c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:03.381624 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acef9c65-9be5-4aa3-8c98-a590b6f87d1c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:03.381903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acef9c65-9be5-4aa3-8c98-a590b6f87d1c None admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:03.383139 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acef9c65-9be5-4aa3-8c98-a590b6f87d1c None admin] RBAC: Authorizing `identity:get_project(project_id=06615629856547fd9da00a208187c681)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:03.384921 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acef9c65-9be5-4aa3-8c98-a590b6f87d1c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:03.386748 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 158/323] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:44:03 2022] GET /identity/v3/projects/06615629856547fd9da00a208187c681 => generated 302 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:03.387578 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acef9c65-9be5-4aa3-8c98-a590b6f87d1c None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:03.418587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ac7a26-1c9f-4e51-8622-169e678cf060 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:03.419623 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ac7a26-1c9f-4e51-8622-169e678cf060 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:03.420080 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ac7a26-1c9f-4e51-8622-169e678cf060 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:03.421634 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ac7a26-1c9f-4e51-8622-169e678cf060 None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:03.423555 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ac7a26-1c9f-4e51-8622-169e678cf060 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:03.423555 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ac7a26-1c9f-4e51-8622-169e678cf060 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:03.425780 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ac7a26-1c9f-4e51-8622-169e678cf060 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:03.446294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 166/324] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:44:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:03.446294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ac7a26-1c9f-4e51-8622-169e678cf060 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:03.535559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:03.550062 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acef9c65-9be5-4aa3-8c98-a590b6f87d1c None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:03.596462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ac7a26-1c9f-4e51-8622-169e678cf060 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:03.598096 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add3a3c5-b821-4cc9-ae36-617f697159d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:03.598846 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add3a3c5-b821-4cc9-ae36-617f697159d0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:03.599315 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add3a3c5-b821-4cc9-ae36-617f697159d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:03.599767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add3a3c5-b821-4cc9-ae36-617f697159d0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:03.612544 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add3a3c5-b821-4cc9-ae36-617f697159d0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:03.617969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-add3a3c5-b821-4cc9-ae36-617f697159d0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:44:03.617969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:03.617969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:03.617969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:03.617969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:03.617969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:03.617969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:03.617969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:03.617969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:03.617969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:03.617969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:03.617969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:03.617969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:03.617969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:03.617969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:03.617969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:03.620012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:03.620012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:03.620012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:03.620012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:03.620012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:03.620012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:03.620012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:03.620012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:03.620012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:03.620012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:03.620012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:03.620012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:03.620012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:03.620012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:03.620012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:03.620012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:03.620833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:03.620833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:03.620833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:03.620833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:03.620833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:03.620833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:03.620833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:03.620833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:03.620833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:03.620833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:03.620833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:44:03.620833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:03.620833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 159/325] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 16 00:44:03 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:03.620833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add3a3c5-b821-4cc9-ae36-617f697159d0 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:03.630237 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:03.639903 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba130666-dc26-4491-953b-8bdd45c61b8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:03.640580 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba130666-dc26-4491-953b-8bdd45c61b8a None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:03.640836 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba130666-dc26-4491-953b-8bdd45c61b8a None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:03.641114 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba130666-dc26-4491-953b-8bdd45c61b8a None admin] PATH_INFO: `/v3/groups/2597fd1ab32f49e1a4978291427c998c` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:03.642289 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba130666-dc26-4491-953b-8bdd45c61b8a None admin] RBAC: Authorizing `identity:get_group(group_id=2597fd1ab32f49e1a4978291427c998c)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:03.643820 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba130666-dc26-4491-953b-8bdd45c61b8a None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:03.645456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 167/326] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 16 00:44:03 2022] GET /identity/v3/groups/2597fd1ab32f49e1a4978291427c998c => generated 226 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:03.646239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba130666-dc26-4491-953b-8bdd45c61b8a None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:03.743351 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add3a3c5-b821-4cc9-ae36-617f697159d0 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:03.756157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba130666-dc26-4491-953b-8bdd45c61b8a None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:03.781834 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b2d961-e812-4ac4-8b7b-37ecdb7d9fd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:03.782452 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b2d961-e812-4ac4-8b7b-37ecdb7d9fd9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:03.782735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b2d961-e812-4ac4-8b7b-37ecdb7d9fd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:03.783069 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b2d961-e812-4ac4-8b7b-37ecdb7d9fd9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:03.797943 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec721078-37dd-4523-9b33-7dac22bca30b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:03.798977 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec721078-37dd-4523-9b33-7dac22bca30b None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:03.799434 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec721078-37dd-4523-9b33-7dac22bca30b None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:03.799842 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec721078-37dd-4523-9b33-7dac22bca30b None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:03.800617 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec721078-37dd-4523-9b33-7dac22bca30b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:03.801014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec721078-37dd-4523-9b33-7dac22bca30b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:03.802653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec721078-37dd-4523-9b33-7dac22bca30b None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:03.811725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:03.811985 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 168/327] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:03 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:03.812854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec721078-37dd-4523-9b33-7dac22bca30b None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:03.871937 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b2d961-e812-4ac4-8b7b-37ecdb7d9fd9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:03.873439 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b2d961-e812-4ac4-8b7b-37ecdb7d9fd9 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:03.883378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:03.939148 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec721078-37dd-4523-9b33-7dac22bca30b None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:03.977699 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952a001b-5e63-4221-be81-8f9a5dd8d2c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:03.978844 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952a001b-5e63-4221-be81-8f9a5dd8d2c9 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:03.979162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952a001b-5e63-4221-be81-8f9a5dd8d2c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:03.979540 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952a001b-5e63-4221-be81-8f9a5dd8d2c9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:03.980282 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952a001b-5e63-4221-be81-8f9a5dd8d2c9 None admin] RBAC: Adding query filter params (group.id=2597fd1ab32f49e1a4978291427c998c, role.id=8a6c7a6c7d154f3db6d2639609d60224, scope.project.id=06615629856547fd9da00a208187c681) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:03.980592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952a001b-5e63-4221-be81-8f9a5dd8d2c9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:03.982053 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952a001b-5e63-4221-be81-8f9a5dd8d2c9 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:03.988281 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 169/328] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Oct 16 00:44:03 2022] GET /identity/v3/role_assignments?group.id=2597fd1ab32f49e1a4978291427c998c&scope.project.id=06615629856547fd9da00a208187c681&role.id=8a6c7a6c7d154f3db6d2639609d60224 => generated 611 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:03.989235 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952a001b-5e63-4221-be81-8f9a5dd8d2c9 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:04.015926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 160/329] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:44:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:44:04.022955 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b2d961-e812-4ac4-8b7b-37ecdb7d9fd9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:04.096684 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:04.133286 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952a001b-5e63-4221-be81-8f9a5dd8d2c9 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:04.167154 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:04.182434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b2d961-e812-4ac4-8b7b-37ecdb7d9fd9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:04.184933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f6ba2b-3c71-4db5-ba9f-aa00a5dfc2d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:04.185724 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f6ba2b-3c71-4db5-ba9f-aa00a5dfc2d1 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:04.186003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f6ba2b-3c71-4db5-ba9f-aa00a5dfc2d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:04.186276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f6ba2b-3c71-4db5-ba9f-aa00a5dfc2d1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:04.253714 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bced0e-3398-4420-910d-fc1d9a40f69f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:04.254617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bced0e-3398-4420-910d-fc1d9a40f69f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:04.255087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bced0e-3398-4420-910d-fc1d9a40f69f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:04.255544 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bced0e-3398-4420-910d-fc1d9a40f69f None admin] PATH_INFO: `/v3/projects/f1d89d5c90cd4c35b26fbcfc82450844` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:04.257949 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bced0e-3398-4420-910d-fc1d9a40f69f None admin] RBAC: Authorizing `identity:get_project(project_id=f1d89d5c90cd4c35b26fbcfc82450844)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:04.260956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bced0e-3398-4420-910d-fc1d9a40f69f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:04.264212 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 161/330] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:44:04 2022] GET /identity/v3/projects/f1d89d5c90cd4c35b26fbcfc82450844 => generated 344 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:04.273018 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bced0e-3398-4420-910d-fc1d9a40f69f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:04.313989 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f6ba2b-3c71-4db5-ba9f-aa00a5dfc2d1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:04.316633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f6ba2b-3c71-4db5-ba9f-aa00a5dfc2d1 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:04.452986 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bced0e-3398-4420-910d-fc1d9a40f69f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:04.484498 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:04.505117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a01e65e-09d3-482c-880e-bacd637ef1f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:04.505873 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a01e65e-09d3-482c-880e-bacd637ef1f8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:04.506213 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a01e65e-09d3-482c-880e-bacd637ef1f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:04.506662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a01e65e-09d3-482c-880e-bacd637ef1f8 None admin] PATH_INFO: `/v3/groups/b7e1cc2052ba47cc9e29d0ad9c80d4bf` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:04.508313 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a01e65e-09d3-482c-880e-bacd637ef1f8 None admin] RBAC: Authorizing `identity:get_group(group_id=b7e1cc2052ba47cc9e29d0ad9c80d4bf)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:04.510787 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a01e65e-09d3-482c-880e-bacd637ef1f8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:04.513056 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 162/331] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 16 00:44:04 2022] GET /identity/v3/groups/b7e1cc2052ba47cc9e29d0ad9c80d4bf => generated 229 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:04.519333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a01e65e-09d3-482c-880e-bacd637ef1f8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:04.520963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 170/332] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:44:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 532 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:44:04.527865 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f6ba2b-3c71-4db5-ba9f-aa00a5dfc2d1 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:04.558044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:04.642660 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a01e65e-09d3-482c-880e-bacd637ef1f8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:04.678475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f6ba2b-3c71-4db5-ba9f-aa00a5dfc2d1 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:04.686644 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abc34ba-1219-45d0-b951-5dde81a41254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:04.687627 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abc34ba-1219-45d0-b951-5dde81a41254 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:04.688101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abc34ba-1219-45d0-b951-5dde81a41254 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:04.688548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abc34ba-1219-45d0-b951-5dde81a41254 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:04.689793 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abc34ba-1219-45d0-b951-5dde81a41254 None admin] RBAC: Adding query filter params (group.id=b7e1cc2052ba47cc9e29d0ad9c80d4bf, role.id=c5cd777d70a649028f970fd986dc484f, scope.project.id=f1d89d5c90cd4c35b26fbcfc82450844) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:04.690321 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abc34ba-1219-45d0-b951-5dde81a41254 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:04.692996 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abc34ba-1219-45d0-b951-5dde81a41254 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:04.704628 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 163/333] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Oct 16 00:44:04 2022] GET /identity/v3/role_assignments?group.id=b7e1cc2052ba47cc9e29d0ad9c80d4bf&scope.project.id=f1d89d5c90cd4c35b26fbcfc82450844&role.id=c5cd777d70a649028f970fd986dc484f => generated 611 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:04.719788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0abf58f-ee52-4327-8953-7764e7e79cea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:04.720479 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0abf58f-ee52-4327-8953-7764e7e79cea None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:04.720762 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0abf58f-ee52-4327-8953-7764e7e79cea None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:04.721038 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0abf58f-ee52-4327-8953-7764e7e79cea None admin] PATH_INFO: `/v3/projects/06615629856547fd9da00a208187c681` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:04.722461 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0abf58f-ee52-4327-8953-7764e7e79cea None admin] RBAC: Authorizing `identity:get_project(project_id=06615629856547fd9da00a208187c681)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:04.724174 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0abf58f-ee52-4327-8953-7764e7e79cea None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:04.726036 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 171/334] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:44:04 2022] GET /identity/v3/projects/06615629856547fd9da00a208187c681 => generated 302 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:04.733374 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abc34ba-1219-45d0-b951-5dde81a41254 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:04.840749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:04.920646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abc34ba-1219-45d0-b951-5dde81a41254 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:04.972736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd50c7b-0733-493d-b8f4-46685591f32b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:04.973655 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd50c7b-0733-493d-b8f4-46685591f32b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:04.974095 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd50c7b-0733-493d-b8f4-46685591f32b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:04.974508 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd50c7b-0733-493d-b8f4-46685591f32b None admin] PATH_INFO: `/v3/groups/2597fd1ab32f49e1a4978291427c998c` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:04.976024 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd50c7b-0733-493d-b8f4-46685591f32b None admin] RBAC: Authorizing `identity:get_group(group_id=2597fd1ab32f49e1a4978291427c998c)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:04.978115 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd50c7b-0733-493d-b8f4-46685591f32b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:04.980811 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 164/335] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 16 00:44:04 2022] GET /identity/v3/groups/2597fd1ab32f49e1a4978291427c998c => generated 226 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:04.988466 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0abf58f-ee52-4327-8953-7764e7e79cea None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:05.096180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:05.108143 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0abf58f-ee52-4327-8953-7764e7e79cea None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:05.148046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f140223c-70a4-449e-882b-297ebcea49a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:05.148684 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f140223c-70a4-449e-882b-297ebcea49a3 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:05.148966 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f140223c-70a4-449e-882b-297ebcea49a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:05.149241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f140223c-70a4-449e-882b-297ebcea49a3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:05.149990 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f140223c-70a4-449e-882b-297ebcea49a3 None admin] RBAC: Adding query filter params (group.id=2597fd1ab32f49e1a4978291427c998c, role.id=d9dcb4cfc0a245a98a7d96c100a497a1, scope.project.id=06615629856547fd9da00a208187c681) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:05.150994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f140223c-70a4-449e-882b-297ebcea49a3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:05.152535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f140223c-70a4-449e-882b-297ebcea49a3 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:05.159389 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 172/336] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Oct 16 00:44:04 2022] GET /identity/v3/role_assignments?group.id=2597fd1ab32f49e1a4978291427c998c&scope.project.id=06615629856547fd9da00a208187c681&role.id=d9dcb4cfc0a245a98a7d96c100a497a1 => generated 611 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:09.023923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3ca224-fb05-4fed-b3f3-b6af814b892f None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:09.025060 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3ca224-fb05-4fed-b3f3-b6af814b892f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:09.025718 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3ca224-fb05-4fed-b3f3-b6af814b892f None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:09.027477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 165/337] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:09 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:44:09.043984 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac6eaad-ee2c-458e-81c2-b8d1c8d5b155 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:09.044981 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac6eaad-ee2c-458e-81c2-b8d1c8d5b155 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:09.045490 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac6eaad-ee2c-458e-81c2-b8d1c8d5b155 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:09.073304 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-aac6eaad-ee2c-458e-81c2-b8d1c8d5b155 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:09.075193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-aac6eaad-ee2c-458e-81c2-b8d1c8d5b155 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:09.226029 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 173/338] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:44:09.329046 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6b4fb0-316f-4ca9-8477-c9512cf44298 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:09.329522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6b4fb0-316f-4ca9-8477-c9512cf44298 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:09.330269 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6b4fb0-316f-4ca9-8477-c9512cf44298 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:09.333153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 166/339] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:09 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:09.343214 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d32c4c-b386-4ad1-9f35-5b11d4a0ced2 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:09.343709 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d32c4c-b386-4ad1-9f35-5b11d4a0ced2 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:09.344200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d32c4c-b386-4ad1-9f35-5b11d4a0ced2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:09.375255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-e3d32c4c-b386-4ad1-9f35-5b11d4a0ced2 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:09.377344 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-e3d32c4c-b386-4ad1-9f35-5b11d4a0ced2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:09.575788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 174/340] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:44:09.966553 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6b4fb0-316f-4ca9-8477-c9512cf44298 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:10.112143 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d32c4c-b386-4ad1-9f35-5b11d4a0ced2 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:10.143289 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:10.202261 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6b4fb0-316f-4ca9-8477-c9512cf44298 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:10.245587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:10.268455 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8107d0a-5914-4577-9d6b-5f1da6a4cb1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:10.269353 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8107d0a-5914-4577-9d6b-5f1da6a4cb1a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:10.269807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8107d0a-5914-4577-9d6b-5f1da6a4cb1a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:10.270222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8107d0a-5914-4577-9d6b-5f1da6a4cb1a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:10.289663 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8107d0a-5914-4577-9d6b-5f1da6a4cb1a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:10.291395 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8107d0a-5914-4577-9d6b-5f1da6a4cb1a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:10.294318 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d32c4c-b386-4ad1-9f35-5b11d4a0ced2 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:10.308177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-f8107d0a-5914-4577-9d6b-5f1da6a4cb1a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:10.308177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:10.308177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:10.308177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:10.308177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:10.308177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:10.308177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:10.308177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:10.308177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:10.308177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:10.308177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:10.308177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:10.308177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:10.308177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:10.308177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:10.308177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:10.309774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:10.309774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:10.309774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:10.309774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:10.309774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:10.309774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:10.309774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:10.309774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:10.309774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:10.309774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:10.309774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:10.309774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:10.309774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:10.309774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:10.309774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:10.309774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:10.309774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:10.311496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:10.311496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:10.311496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:10.311496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 167/341] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 344 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:10.311496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a55c8e3-2910-4448-bfc5-353e789ff2e2 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:10.311496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a55c8e3-2910-4448-bfc5-353e789ff2e2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:10.311496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a55c8e3-2910-4448-bfc5-353e789ff2e2 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:10.312699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 168/342] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:10 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:10.319113 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21d1dcc-949f-4f55-9250-b17d7af09c14 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:10.319113 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21d1dcc-949f-4f55-9250-b17d7af09c14 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:10.319113 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21d1dcc-949f-4f55-9250-b17d7af09c14 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:10.319113 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 169/343] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:10 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:10.319113 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b40b3b3-3c28-4d0d-bff2-40c1c9ba6810 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:10.319113 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b40b3b3-3c28-4d0d-bff2-40c1c9ba6810 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:10.319931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b40b3b3-3c28-4d0d-bff2-40c1c9ba6810 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:10.319931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 170/344] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:10.321438 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b40b3b3-3c28-4d0d-bff2-40c1c9ba6810 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:10.338811 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd0b90f-e138-4b5e-979c-789844a5b595 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:10.339626 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd0b90f-e138-4b5e-979c-789844a5b595 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:10.339969 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd0b90f-e138-4b5e-979c-789844a5b595 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:10.340305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd0b90f-e138-4b5e-979c-789844a5b595 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:10.352605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd0b90f-e138-4b5e-979c-789844a5b595 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:10.354802 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd0b90f-e138-4b5e-979c-789844a5b595 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:10.376028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-cdd0b90f-e138-4b5e-979c-789844a5b595 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:10.376028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:10.376028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:10.376028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:10.376028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:10.376028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:10.376028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:10.376028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:10.376028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:10.376028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:10.376028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:10.376028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:10.376028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:10.376028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:10.376028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:10.376028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:10.377030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:10.377030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:10.377030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:10.377030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:10.377030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:10.377030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:10.377030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:10.377030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:10.377030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:10.377030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:10.377030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:10.377030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:10.377030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:10.377030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:10.377030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:10.377030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:10.377030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:10.377901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:10.377901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:10.377901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:10.377901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 175/345] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:10.379961 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13c6f85-5544-4084-a2b9-6bc5e24d162c None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:10.380262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13c6f85-5544-4084-a2b9-6bc5e24d162c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:10.380542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13c6f85-5544-4084-a2b9-6bc5e24d162c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:10.398944 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-e13c6f85-5544-4084-a2b9-6bc5e24d162c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:10.400724 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-e13c6f85-5544-4084-a2b9-6bc5e24d162c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:10.463642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b40b3b3-3c28-4d0d-bff2-40c1c9ba6810 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:10.473789 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:10.486571 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 176/346] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:10.487682 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b4128e-ee58-430b-b18b-a61287963ec5 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:10.488018 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b4128e-ee58-430b-b18b-a61287963ec5 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:10.488247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b4128e-ee58-430b-b18b-a61287963ec5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:10.510362 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-42b4128e-ee58-430b-b18b-a61287963ec5 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:10.511686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-42b4128e-ee58-430b-b18b-a61287963ec5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:10.513274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402bdabf-ca52-43e3-bbed-234a8f8b4fd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:10.514195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402bdabf-ca52-43e3-bbed-234a8f8b4fd4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:10.514623 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402bdabf-ca52-43e3-bbed-234a8f8b4fd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:10.515076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402bdabf-ca52-43e3-bbed-234a8f8b4fd4 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:10.515988 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402bdabf-ca52-43e3-bbed-234a8f8b4fd4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:10.516314 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402bdabf-ca52-43e3-bbed-234a8f8b4fd4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:10.517728 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402bdabf-ca52-43e3-bbed-234a8f8b4fd4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:10.545169 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 171/347] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:10.545169 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a95c2d6-c7bd-42f0-9cc4-d2f9deb2bf7c None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:10.545169 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a95c2d6-c7bd-42f0-9cc4-d2f9deb2bf7c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:10.545169 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a95c2d6-c7bd-42f0-9cc4-d2f9deb2bf7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:10.564092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2a95c2d6-c7bd-42f0-9cc4-d2f9deb2bf7c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:10.565297 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2a95c2d6-c7bd-42f0-9cc4-d2f9deb2bf7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:10.608244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:10.626903 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 177/348] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:10.626903 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b4128e-ee58-430b-b18b-a61287963ec5 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:10.680632 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 172/349] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:10.682633 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a95c2d6-c7bd-42f0-9cc4-d2f9deb2bf7c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:10.825529 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b4128e-ee58-430b-b18b-a61287963ec5 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:10.878166 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:10.886404 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a95c2d6-c7bd-42f0-9cc4-d2f9deb2bf7c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:10.893343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea768aa-6c37-4706-a264-fd988f8fd066 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:10.894280 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea768aa-6c37-4706-a264-fd988f8fd066 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:10.894778 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea768aa-6c37-4706-a264-fd988f8fd066 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:10.895231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea768aa-6c37-4706-a264-fd988f8fd066 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:10.896190 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea768aa-6c37-4706-a264-fd988f8fd066 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:10.896807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea768aa-6c37-4706-a264-fd988f8fd066 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:10.898582 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea768aa-6c37-4706-a264-fd988f8fd066 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:10.923071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 178/350] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:10.924082 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea768aa-6c37-4706-a264-fd988f8fd066 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:10.938693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca8cb48-5a27-4c3d-846b-359345d0fa3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:10.939365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca8cb48-5a27-4c3d-846b-359345d0fa3f None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:10.939649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca8cb48-5a27-4c3d-846b-359345d0fa3f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:10.939928 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca8cb48-5a27-4c3d-846b-359345d0fa3f None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:10.940606 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca8cb48-5a27-4c3d-846b-359345d0fa3f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:10.942123 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca8cb48-5a27-4c3d-846b-359345d0fa3f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:10.978918 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 173/351] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:44:10 2022] POST /identity/v3/users => generated 246 bytes in 291 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:44:10.978918 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca8cb48-5a27-4c3d-846b-359345d0fa3f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:11.011324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:11.104794 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea768aa-6c37-4706-a264-fd988f8fd066 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:11.161270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d50dc7-2cb0-4953-aab4-8a9a0f89a80f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:11.161930 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d50dc7-2cb0-4953-aab4-8a9a0f89a80f None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:11.162315 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d50dc7-2cb0-4953-aab4-8a9a0f89a80f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:11.162654 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d50dc7-2cb0-4953-aab4-8a9a0f89a80f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:11.162745 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca8cb48-5a27-4c3d-846b-359345d0fa3f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:11.181293 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d50dc7-2cb0-4953-aab4-8a9a0f89a80f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:11.182980 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d50dc7-2cb0-4953-aab4-8a9a0f89a80f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:11.211056 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3059faa-dc3e-4ece-988e-5a592954b478 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:11.212284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3059faa-dc3e-4ece-988e-5a592954b478 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:11.213106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-54d50dc7-2cb0-4953-aab4-8a9a0f89a80f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:11.213106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:11.213106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:11.213106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:11.213106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:11.213106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:11.213106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:11.213106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:11.213106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:11.213106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:11.213106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:11.213106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:11.213106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:11.213106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:11.213106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:11.213106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:11.214317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:11.214317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:11.214317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:11.214317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:11.214317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:11.214317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:11.214317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:11.214317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:11.214317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:11.214317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:11.214317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:11.214317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:11.214317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:11.214317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:11.214317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:11.214317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:11.214317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:11.215288 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:11.215288 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:11.215288 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:11.215288 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 179/352] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:11.215918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d50dc7-2cb0-4953-aab4-8a9a0f89a80f None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:11.216339 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3059faa-dc3e-4ece-988e-5a592954b478 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:11.216749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3059faa-dc3e-4ece-988e-5a592954b478 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:11.231052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3059faa-dc3e-4ece-988e-5a592954b478 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:11.233267 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3059faa-dc3e-4ece-988e-5a592954b478 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:11.251341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-e3059faa-dc3e-4ece-988e-5a592954b478 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:11.251341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:11.251341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:11.251341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:11.251341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:11.251341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:11.251341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:11.251341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:11.251341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:11.251341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:11.251341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:11.251341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:11.251341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:11.251341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:11.251341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:11.251341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:11.252324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:11.252324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:11.252324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:11.252324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:11.252324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:11.252324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:11.252324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:11.252324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:11.252324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:11.252324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:11.252324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:11.252324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:11.252324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:11.252324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:11.252324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:11.252324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:11.252324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:11.253814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:11.253814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:11.253814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:11.253814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 174/353] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:11.253814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3059faa-dc3e-4ece-988e-5a592954b478 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:11.287077 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:11.371811 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:11.374629 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d50dc7-2cb0-4953-aab4-8a9a0f89a80f None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:11.412900 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3059faa-dc3e-4ece-988e-5a592954b478 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:11.437363 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d4c29f-7b79-40d2-a14f-ee5e7714bc44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:11.437998 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d4c29f-7b79-40d2-a14f-ee5e7714bc44 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:11.438272 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d4c29f-7b79-40d2-a14f-ee5e7714bc44 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:11.438548 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d4c29f-7b79-40d2-a14f-ee5e7714bc44 None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:11.439231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d4c29f-7b79-40d2-a14f-ee5e7714bc44 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:11.440686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d4c29f-7b79-40d2-a14f-ee5e7714bc44 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:11.464791 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497497fd-1ff6-4694-ad3b-39c7542fde22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:11.473110 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 180/354] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:44:11 2022] POST /identity/v3/users => generated 248 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:44:11.473332 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497497fd-1ff6-4694-ad3b-39c7542fde22 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:11.473332 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497497fd-1ff6-4694-ad3b-39c7542fde22 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:11.473332 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497497fd-1ff6-4694-ad3b-39c7542fde22 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:11.473558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d4c29f-7b79-40d2-a14f-ee5e7714bc44 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:11.473629 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497497fd-1ff6-4694-ad3b-39c7542fde22 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:11.473629 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497497fd-1ff6-4694-ad3b-39c7542fde22 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:11.480573 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.sql.core [None req-497497fd-1ff6-4694-ad3b-39c7542fde22 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 16 00:44:11.480573 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 16 00:44:11.480573 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [parameters: {'id': '7f4a059d983f4d678adbe6c7c33bdfcd', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 16 00:44:11.480573 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81485) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 16 00:44:11.497540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-497497fd-1ff6-4694-ad3b-39c7542fde22 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 16 00:44:11.497540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:11.497540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 16 00:44:11.497540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 16 00:44:11.497540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 16 00:44:11.497540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 16 00:44:11.497540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 16 00:44:11.497540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result = self._query(query) Oct 16 00:44:11.497540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 16 00:44:11.497540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application conn.query(q) Oct 16 00:44:11.497540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 16 00:44:11.497540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 16 00:44:11.497540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 16 00:44:11.497540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result.read() Oct 16 00:44:11.497540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return ref.to_dict() Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 16 00:44:11.498510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application next(self.gen) Oct 16 00:44:11.499483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 16 00:44:11.499483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application yield resource Oct 16 00:44:11.499483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 16 00:44:11.499483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application next(self.gen) Oct 16 00:44:11.499483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 16 00:44:11.499483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.session.rollback() Oct 16 00:44:11.499483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 16 00:44:11.499483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.force_reraise() Oct 16 00:44:11.499483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 16 00:44:11.499483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise self.value Oct 16 00:44:11.499483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 16 00:44:11.499483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 16 00:44:11.499483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 16 00:44:11.499483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application session.commit() Oct 16 00:44:11.499483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 16 00:44:11.499483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 16 00:44:11.499483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 16 00:44:11.500936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._prepare_impl() Oct 16 00:44:11.500936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 16 00:44:11.500936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.session.flush() Oct 16 00:44:11.500936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 16 00:44:11.500936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._flush(objects) Oct 16 00:44:11.500936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 16 00:44:11.500936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 16 00:44:11.500936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 16 00:44:11.500936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application compat.raise_( Oct 16 00:44:11.500936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 16 00:44:11.500936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception Oct 16 00:44:11.500936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 16 00:44:11.500936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application flush_context.execute() Oct 16 00:44:11.500936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 16 00:44:11.500936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rec.execute(self) Oct 16 00:44:11.500936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 16 00:44:11.500936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 16 00:44:11.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 16 00:44:11.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 16 00:44:11.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 16 00:44:11.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 16 00:44:11.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 16 00:44:11.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 16 00:44:11.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 16 00:44:11.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 16 00:44:11.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 16 00:44:11.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 16 00:44:11.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 16 00:44:11.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 16 00:44:11.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 16 00:44:11.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 16 00:44:11.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 16 00:44:11.501816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception Oct 16 00:44:11.502637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 16 00:44:11.502637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 16 00:44:11.502637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 16 00:44:11.502637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 16 00:44:11.502637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 16 00:44:11.502637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result = self._query(query) Oct 16 00:44:11.502637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 16 00:44:11.502637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application conn.query(q) Oct 16 00:44:11.502637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 16 00:44:11.502637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 16 00:44:11.502637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 16 00:44:11.502637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result.read() Oct 16 00:44:11.502637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 16 00:44:11.502637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 16 00:44:11.502637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 16 00:44:11.502637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 16 00:44:11.508611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 16 00:44:11.508611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 16 00:44:11.508611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 16 00:44:11.508611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 16 00:44:11.508611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 16 00:44:11.508611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 16 00:44:11.508611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [parameters: {'id': '7f4a059d983f4d678adbe6c7c33bdfcd', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 16 00:44:11.508611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 16 00:44:11.508611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:11.508611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:44:11.508611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:11.508611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:11.508611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:11.508611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:11.508611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:11.508611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:11.511232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:11.511232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:11.511232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:11.511232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:11.511232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:11.511232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:11.511232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 16 00:44:11.511232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 16 00:44:11.511232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:11.511232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:11.511232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 16 00:44:11.511232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 16 00:44:11.511232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 16 00:44:11.511232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 16 00:44:11.511232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 16 00:44:11.511232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:11.513648 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 175/355] 10.222.0.25 () {60 vars in 1131 bytes} [Sun Oct 16 00:44:11 2022] POST /identity/v3/roles => generated 144 bytes in 250 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:11.513648 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497497fd-1ff6-4694-ad3b-39c7542fde22 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:11.605621 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:11.613560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d4c29f-7b79-40d2-a14f-ee5e7714bc44 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:11.652808 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497497fd-1ff6-4694-ad3b-39c7542fde22 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:11.671601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398515a6-a1b7-47db-8221-609ce2c1d7f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:11.672625 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398515a6-a1b7-47db-8221-609ce2c1d7f7 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:11.673247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398515a6-a1b7-47db-8221-609ce2c1d7f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:11.673721 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398515a6-a1b7-47db-8221-609ce2c1d7f7 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:11.674701 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398515a6-a1b7-47db-8221-609ce2c1d7f7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:11.674933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398515a6-a1b7-47db-8221-609ce2c1d7f7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:11.676291 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398515a6-a1b7-47db-8221-609ce2c1d7f7 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:11.692560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 181/356] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:11.693170 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398515a6-a1b7-47db-8221-609ce2c1d7f7 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:11.708887 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e852a7-9826-438d-ae0d-8be1a6a29a63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:11.709652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e852a7-9826-438d-ae0d-8be1a6a29a63 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:11.709904 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e852a7-9826-438d-ae0d-8be1a6a29a63 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:11.710188 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e852a7-9826-438d-ae0d-8be1a6a29a63 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:11.710917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e852a7-9826-438d-ae0d-8be1a6a29a63 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:11.711299 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e852a7-9826-438d-ae0d-8be1a6a29a63 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:11.714052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e852a7-9826-438d-ae0d-8be1a6a29a63 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:11.716889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:11.733698 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 176/357] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:11.735334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e852a7-9826-438d-ae0d-8be1a6a29a63 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:11.855280 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398515a6-a1b7-47db-8221-609ce2c1d7f7 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:11.926064 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f262d26-be1d-4bb6-948d-6fe8c9e97f61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:11.927077 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f262d26-be1d-4bb6-948d-6fe8c9e97f61 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:11.927538 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f262d26-be1d-4bb6-948d-6fe8c9e97f61 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:11.927980 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f262d26-be1d-4bb6-948d-6fe8c9e97f61 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:11.929390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e852a7-9826-438d-ae0d-8be1a6a29a63 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:11.941906 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f262d26-be1d-4bb6-948d-6fe8c9e97f61 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:11.948109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-7f262d26-be1d-4bb6-948d-6fe8c9e97f61 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:44:11.948109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:11.948109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:11.948109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:11.948109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:11.948109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:11.948109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:11.948109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:11.948109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:11.948109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:11.948109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:11.948109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:11.948109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:11.948109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:11.948109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:11.948109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:11.949309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:11.949309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:11.949309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:11.949309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:11.949309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:11.949309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:11.949309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:11.949309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:11.949309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:11.949309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:11.949309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:11.949309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:11.949309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:11.949309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:11.949309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:11.949309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:11.950193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:11.950193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:11.950193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:11.950193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:11.950193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:11.950193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:11.950193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:11.950193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:11.950193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:11.950193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:11.950193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 16 00:44:11.950193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:11.952290 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 182/358] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 16 00:44:11 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:11.958558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f262d26-be1d-4bb6-948d-6fe8c9e97f61 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:11.980238 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:12.011460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dda468-08b0-42f8-920d-13db6b5efd2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:12.012875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dda468-08b0-42f8-920d-13db6b5efd2d None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:12.013598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dda468-08b0-42f8-920d-13db6b5efd2d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:12.014299 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dda468-08b0-42f8-920d-13db6b5efd2d None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:12.015277 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85dda468-08b0-42f8-920d-13db6b5efd2d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:12.018009 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85dda468-08b0-42f8-920d-13db6b5efd2d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:12.053216 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 177/359] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:44:11 2022] POST /identity/v3/users => generated 248 bytes in 318 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:44:12.053910 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dda468-08b0-42f8-920d-13db6b5efd2d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:12.135385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:12.175577 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f262d26-be1d-4bb6-948d-6fe8c9e97f61 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:12.275809 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dda468-08b0-42f8-920d-13db6b5efd2d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:12.279046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b5f122-eb5e-4d2c-9c72-0ad7c2970d36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:12.279688 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b5f122-eb5e-4d2c-9c72-0ad7c2970d36 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:12.279966 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b5f122-eb5e-4d2c-9c72-0ad7c2970d36 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:12.280237 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b5f122-eb5e-4d2c-9c72-0ad7c2970d36 None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:12.282106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b5f122-eb5e-4d2c-9c72-0ad7c2970d36 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:12.288111 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b5f122-eb5e-4d2c-9c72-0ad7c2970d36 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:12.330890 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 183/360] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:44:11 2022] POST /identity/v3/users => generated 249 bytes in 373 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:44:12.338323 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309d9e83-0b01-42fa-afba-7e48600d9ae8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:12.338770 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309d9e83-0b01-42fa-afba-7e48600d9ae8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:12.339087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309d9e83-0b01-42fa-afba-7e48600d9ae8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:12.339371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309d9e83-0b01-42fa-afba-7e48600d9ae8 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:12.340107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309d9e83-0b01-42fa-afba-7e48600d9ae8 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:12.340424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309d9e83-0b01-42fa-afba-7e48600d9ae8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:12.341982 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309d9e83-0b01-42fa-afba-7e48600d9ae8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:12.352765 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 178/361] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:12 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:13.965035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e045af4-5e45-4f9e-b8ec-f146e1b5e9ff None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:13.966570 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e045af4-5e45-4f9e-b8ec-f146e1b5e9ff None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:13.967396 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e045af4-5e45-4f9e-b8ec-f146e1b5e9ff None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:13.970150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 184/362] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:13 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:13.979603 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf725179-4827-429e-9d54-c99d580c5a97 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:13.981046 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf725179-4827-429e-9d54-c99d580c5a97 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:13.981803 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf725179-4827-429e-9d54-c99d580c5a97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:14.014030 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-bf725179-4827-429e-9d54-c99d580c5a97 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:14.016503 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bf725179-4827-429e-9d54-c99d580c5a97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:14.168525 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 179/363] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:14.811765 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e045af4-5e45-4f9e-b8ec-f146e1b5e9ff None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:14.940433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:14.987213 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e045af4-5e45-4f9e-b8ec-f146e1b5e9ff None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:15.034320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23700e04-c90f-4a46-bb1a-1f8a1164f8af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:15.035005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23700e04-c90f-4a46-bb1a-1f8a1164f8af None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:15.035284 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23700e04-c90f-4a46-bb1a-1f8a1164f8af None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:15.035563 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23700e04-c90f-4a46-bb1a-1f8a1164f8af None admin] PATH_INFO: `/v3/roles/service` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:15.051859 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23700e04-c90f-4a46-bb1a-1f8a1164f8af None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:15.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-23700e04-c90f-4a46-bb1a-1f8a1164f8af None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:15.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:15.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:15.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:15.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:15.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:15.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:15.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:15.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:15.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:15.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:15.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:15.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:15.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:15.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:15.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:15.059668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:15.059668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:15.059668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:15.059668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:15.059668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:15.059668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:15.059668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:15.059668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:15.059668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:15.059668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:15.059668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:15.059668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:15.059668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:15.059668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:15.059668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:15.059668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:15.061239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:15.061239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:15.061239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:15.061239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:15.061239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:15.061239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:15.061239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:15.061239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:15.061239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:15.061239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:15.061239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:15.061239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:15.061239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 185/364] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 16 00:44:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:15.072612 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23700e04-c90f-4a46-bb1a-1f8a1164f8af None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:15.073932 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881ee6a5-5b9a-4876-88f1-2faccc1bfd9e None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:15.075280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881ee6a5-5b9a-4876-88f1-2faccc1bfd9e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:15.076078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881ee6a5-5b9a-4876-88f1-2faccc1bfd9e None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:15.083365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 180/365] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:15 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:15.088577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6932901c-9d73-43d2-ae33-a60fcea9d29f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:15.089410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6932901c-9d73-43d2-ae33-a60fcea9d29f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:15.090109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6932901c-9d73-43d2-ae33-a60fcea9d29f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:15.121786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-6932901c-9d73-43d2-ae33-a60fcea9d29f None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:15.124525 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6932901c-9d73-43d2-ae33-a60fcea9d29f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:15.211679 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:15.273508 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23700e04-c90f-4a46-bb1a-1f8a1164f8af None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:15.277033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 181/366] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:44:15.280535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6753d4-7aa5-45f1-b5d4-c36fef2cb9b4 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:15.281307 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6753d4-7aa5-45f1-b5d4-c36fef2cb9b4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:15.282022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6753d4-7aa5-45f1-b5d4-c36fef2cb9b4 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:15.284600 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 182/367] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:15 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:15.290822 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2712ad-c504-44c6-890c-da6e040ebd9a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:15.291139 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2712ad-c504-44c6-890c-da6e040ebd9a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:15.291578 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2712ad-c504-44c6-890c-da6e040ebd9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:15.335496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-7e2712ad-c504-44c6-890c-da6e040ebd9a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:15.336944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7e2712ad-c504-44c6-890c-da6e040ebd9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:15.346908 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:15.362971 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665d61e7-de71-4c1b-b0fb-60b82e15dc29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:15.364081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665d61e7-de71-4c1b-b0fb-60b82e15dc29 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:15.364578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665d61e7-de71-4c1b-b0fb-60b82e15dc29 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:15.365040 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665d61e7-de71-4c1b-b0fb-60b82e15dc29 None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:15.366290 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665d61e7-de71-4c1b-b0fb-60b82e15dc29 None admin] RBAC: Adding query filter params (name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:15.366911 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665d61e7-de71-4c1b-b0fb-60b82e15dc29 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:15.370055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665d61e7-de71-4c1b-b0fb-60b82e15dc29 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:15.402363 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 186/368] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 16 00:44:15 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 330 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:15.414681 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665d61e7-de71-4c1b-b0fb-60b82e15dc29 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:15.564792 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 183/369] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:15.632672 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665d61e7-de71-4c1b-b0fb-60b82e15dc29 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:15.695702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2712ad-c504-44c6-890c-da6e040ebd9a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:15.716006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bde6411-bdae-4f04-b6ae-3891e2ed647c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:15.716935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bde6411-bdae-4f04-b6ae-3891e2ed647c None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:15.717424 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bde6411-bdae-4f04-b6ae-3891e2ed647c None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:15.717910 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bde6411-bdae-4f04-b6ae-3891e2ed647c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:15.758888 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bde6411-bdae-4f04-b6ae-3891e2ed647c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:15.764677 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bde6411-bdae-4f04-b6ae-3891e2ed647c None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:15.818061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-0bde6411-bdae-4f04-b6ae-3891e2ed647c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:15.818061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:15.818061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:15.818061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:15.818061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:15.818061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:15.818061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:15.818061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:15.818061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:15.818061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:15.818061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:15.818061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:15.818061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:15.818061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:15.818061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:15.818061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:15.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:15.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:15.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:15.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:15.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:15.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:15.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:15.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:15.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:15.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:15.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:15.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:15.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:15.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:15.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:15.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:15.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:15.819986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:15.819986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:15.819986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:15.828864 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 187/370] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 415 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:15.839126 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bde6411-bdae-4f04-b6ae-3891e2ed647c None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:15.963603 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2712ad-c504-44c6-890c-da6e040ebd9a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:15.966825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:15.974476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:16.037804 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2df9e7a-ac77-45b3-af14-8512c83213fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:16.039475 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2df9e7a-ac77-45b3-af14-8512c83213fe None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:16.041174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2df9e7a-ac77-45b3-af14-8512c83213fe None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:16.041949 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2df9e7a-ac77-45b3-af14-8512c83213fe None admin] PATH_INFO: `/v3/roles/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:16.068082 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2df9e7a-ac77-45b3-af14-8512c83213fe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:16.081768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-f2df9e7a-ac77-45b3-af14-8512c83213fe None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:16.081768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:16.081768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:16.081768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:16.081768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:16.081768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:16.081768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:16.081768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:16.081768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:16.081768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:16.081768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:16.081768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:16.081768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:16.081768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:16.081768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:16.081768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:16.084691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:16.084691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:16.084691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:16.084691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:16.084691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:16.084691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:16.084691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:16.084691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:16.084691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:16.084691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:16.084691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:16.084691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:16.084691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:16.084691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:16.084691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:16.084691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:16.086509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:16.086509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:16.086509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:16.086509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:16.086509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:16.086509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:16.086509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:16.086509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:16.086509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:16.086509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:16.086509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:16.086509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:16.091492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 184/371] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 16 00:44:15 2022] GET /identity/v3/roles/service => generated 85 bytes in 396 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:16.093152 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2df9e7a-ac77-45b3-af14-8512c83213fe None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:16.094381 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bde6411-bdae-4f04-b6ae-3891e2ed647c None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:16.162693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbaa4e53-9788-4f94-858c-f57e132d5ae2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:16.163928 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaa4e53-9788-4f94-858c-f57e132d5ae2 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:16.164437 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaa4e53-9788-4f94-858c-f57e132d5ae2 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:16.165024 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaa4e53-9788-4f94-858c-f57e132d5ae2 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:16.166466 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbaa4e53-9788-4f94-858c-f57e132d5ae2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:16.167053 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbaa4e53-9788-4f94-858c-f57e132d5ae2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:16.169885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbaa4e53-9788-4f94-858c-f57e132d5ae2 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:16.199513 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 188/372] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 358 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:16.199513 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf305b5-2f09-4578-97ad-6209f5d1e5dc None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:16.199796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf305b5-2f09-4578-97ad-6209f5d1e5dc None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:16.200360 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf305b5-2f09-4578-97ad-6209f5d1e5dc None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:16.207072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 189/373] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:16 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:16.207072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf305b5-2f09-4578-97ad-6209f5d1e5dc None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:16.270359 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2df9e7a-ac77-45b3-af14-8512c83213fe None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:16.323142 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf305b5-2f09-4578-97ad-6209f5d1e5dc None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:16.325782 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:16.328472 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab17431-da86-4199-a79f-636535e6ff64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:16.329512 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab17431-da86-4199-a79f-636535e6ff64 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:16.330227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab17431-da86-4199-a79f-636535e6ff64 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:16.330695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab17431-da86-4199-a79f-636535e6ff64 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:16.344874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab17431-da86-4199-a79f-636535e6ff64 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:16.349814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-cab17431-da86-4199-a79f-636535e6ff64 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:16.349814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:16.349814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:16.349814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:16.349814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:16.349814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:16.349814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:16.349814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:16.349814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:16.349814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:16.349814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:16.349814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:16.349814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:16.349814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:16.349814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:16.349814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:16.351879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:16.351879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:16.351879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:16.351879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:16.351879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:16.351879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:16.351879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:16.351879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:16.351879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:16.351879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:16.351879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:16.351879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:16.351879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:16.351879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:16.351879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:16.351879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:16.353837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:16.353837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:16.353837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:16.353837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:16.353837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:16.353837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:16.353837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:16.353837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:16.353837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:16.353837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:16.353837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:16.353837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:16.353837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 185/374] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 16 00:44:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:16.353837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab17431-da86-4199-a79f-636535e6ff64 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:16.370962 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9d1784-9fe9-40eb-9d10-3aa481ea37b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:16.371758 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9d1784-9fe9-40eb-9d10-3aa481ea37b8 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:16.372126 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9d1784-9fe9-40eb-9d10-3aa481ea37b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:16.372564 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9d1784-9fe9-40eb-9d10-3aa481ea37b8 None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:16.373455 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9d1784-9fe9-40eb-9d10-3aa481ea37b8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:16.373834 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9d1784-9fe9-40eb-9d10-3aa481ea37b8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:16.375512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9d1784-9fe9-40eb-9d10-3aa481ea37b8 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:16.382299 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:16.388102 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 190/375] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 16 00:44:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:16.389734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fa875c-3cbe-4231-a3f5-cc4e0f2653a5 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:16.390076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fa875c-3cbe-4231-a3f5-cc4e0f2653a5 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:16.390393 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fa875c-3cbe-4231-a3f5-cc4e0f2653a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:16.417305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-26fa875c-3cbe-4231-a3f5-cc4e0f2653a5 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:16.419127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-26fa875c-3cbe-4231-a3f5-cc4e0f2653a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:16.497511 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab17431-da86-4199-a79f-636535e6ff64 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:16.507399 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 191/376] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:16.520914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a02f99-9e58-4920-9e2a-b50180ce0b26 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:16.520914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a02f99-9e58-4920-9e2a-b50180ce0b26 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:16.520914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a02f99-9e58-4920-9e2a-b50180ce0b26 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:16.520914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 192/377] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:16 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:16.520914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a02f99-9e58-4920-9e2a-b50180ce0b26 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:16.548793 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40674670-0b03-4523-9346-26c4bdfae918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:16.549598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40674670-0b03-4523-9346-26c4bdfae918 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:16.550054 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40674670-0b03-4523-9346-26c4bdfae918 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:16.550462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40674670-0b03-4523-9346-26c4bdfae918 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:16.559736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40674670-0b03-4523-9346-26c4bdfae918 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:16.561517 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40674670-0b03-4523-9346-26c4bdfae918 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:16.578682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-40674670-0b03-4523-9346-26c4bdfae918 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 16 00:44:16.578682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:16.578682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:16.578682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:16.578682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:16.578682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:16.578682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:16.578682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:16.578682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:16.578682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:16.578682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:16.578682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:16.578682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:44:16.578682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:44:16.578682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:44:16.578682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:16.581087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:16.582812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:16.582812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:16.582812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:44:16.582812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:44:16.582812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:44:16.582812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:44:16.582812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:44:16.582812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:44:16.582812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 16 00:44:16.582812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:16.582812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 186/378] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Oct 16 00:44:16 2022] GET /identity/v3/users/nova => generated 82 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:16.582812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40674670-0b03-4523-9346-26c4bdfae918 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:16.675755 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:16.688009 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a02f99-9e58-4920-9e2a-b50180ce0b26 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:16.741928 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9f4019-e61b-4eeb-9405-547859b5ab8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:16.742920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9f4019-e61b-4eeb-9405-547859b5ab8f None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:16.743326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9f4019-e61b-4eeb-9405-547859b5ab8f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:16.744527 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9f4019-e61b-4eeb-9405-547859b5ab8f None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:16.745389 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9f4019-e61b-4eeb-9405-547859b5ab8f None admin] RBAC: Adding query filter params (name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:16.745900 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9f4019-e61b-4eeb-9405-547859b5ab8f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:16.747681 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9f4019-e61b-4eeb-9405-547859b5ab8f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:16.750311 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:16.766422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 193/379] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 16 00:44:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:16.766422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62525085-9cf7-40a6-ae82-b494088ff4a5 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:16.766422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62525085-9cf7-40a6-ae82-b494088ff4a5 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:16.766422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62525085-9cf7-40a6-ae82-b494088ff4a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:16.789172 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-62525085-9cf7-40a6-ae82-b494088ff4a5 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:16.789774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40674670-0b03-4523-9346-26c4bdfae918 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:16.791580 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-62525085-9cf7-40a6-ae82-b494088ff4a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:16.851660 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca1edfc-b0ee-4449-91d5-17d3ceea28c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:16.852431 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca1edfc-b0ee-4449-91d5-17d3ceea28c0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:16.852851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca1edfc-b0ee-4449-91d5-17d3ceea28c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:16.853255 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca1edfc-b0ee-4449-91d5-17d3ceea28c0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:16.865724 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca1edfc-b0ee-4449-91d5-17d3ceea28c0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:16.867312 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca1edfc-b0ee-4449-91d5-17d3ceea28c0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:16.885319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-0ca1edfc-b0ee-4449-91d5-17d3ceea28c0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:16.885319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:16.885319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:16.885319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:16.885319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:16.885319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:16.885319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:16.885319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:16.885319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:16.885319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:16.885319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:16.885319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:16.885319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:16.885319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:16.885319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:16.885319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:16.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:16.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:16.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:16.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:16.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:16.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:16.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:16.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:16.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:16.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:16.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:16.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:16.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:16.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:16.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:16.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:16.887396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:16.889078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:16.889078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:16.889078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:16.889078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 187/380] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:16.891515 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca1edfc-b0ee-4449-91d5-17d3ceea28c0 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:16.907130 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 194/381] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:16.908029 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62525085-9cf7-40a6-ae82-b494088ff4a5 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:17.029363 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:17.041908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca1edfc-b0ee-4449-91d5-17d3ceea28c0 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:17.049549 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62525085-9cf7-40a6-ae82-b494088ff4a5 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:17.098141 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48400cbd-857c-4814-8b7c-d34b6ed352b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:17.099199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48400cbd-857c-4814-8b7c-d34b6ed352b1 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:17.099550 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d709c4-da31-4cb4-9680-47cf2e559d4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:17.099929 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48400cbd-857c-4814-8b7c-d34b6ed352b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:17.100144 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48400cbd-857c-4814-8b7c-d34b6ed352b1 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:17.100252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d709c4-da31-4cb4-9680-47cf2e559d4f None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:17.100462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d709c4-da31-4cb4-9680-47cf2e559d4f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:17.100730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d709c4-da31-4cb4-9680-47cf2e559d4f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:17.101539 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48400cbd-857c-4814-8b7c-d34b6ed352b1 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:17.102104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48400cbd-857c-4814-8b7c-d34b6ed352b1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:17.105037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48400cbd-857c-4814-8b7c-d34b6ed352b1 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:17.109169 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:17.113483 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d709c4-da31-4cb4-9680-47cf2e559d4f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:17.115383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d709c4-da31-4cb4-9680-47cf2e559d4f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:17.132341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-14d709c4-da31-4cb4-9680-47cf2e559d4f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:17.132341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:17.132341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:17.132341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:17.132341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:17.132341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:17.132341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:17.132341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:17.132341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:17.132341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:17.132341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:17.132341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:17.132341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:17.132341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:17.132341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:17.132341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:17.133467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:17.133467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:17.133467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:17.133467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:17.133467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:17.133467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:17.133467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:17.133467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:17.133467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:17.133467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:17.133467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:17.133467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:17.133467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:17.133467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:17.133467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:17.133467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:17.133467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:17.135010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:17.135010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:17.135010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:17.135010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 195/382] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:17.135279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d709c4-da31-4cb4-9680-47cf2e559d4f None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:17.170989 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 188/383] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Oct 16 00:44:16 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:17.172521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48400cbd-857c-4814-8b7c-d34b6ed352b1 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:17.298825 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d709c4-da31-4cb4-9680-47cf2e559d4f None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:17.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48400cbd-857c-4814-8b7c-d34b6ed352b1 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:17.360396 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6decb6-710f-4787-9deb-c4bb1cc6381b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:17.361534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6decb6-710f-4787-9deb-c4bb1cc6381b None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:17.362013 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6decb6-710f-4787-9deb-c4bb1cc6381b None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:17.362488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6decb6-710f-4787-9deb-c4bb1cc6381b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:17.377025 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:17.382393 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6decb6-710f-4787-9deb-c4bb1cc6381b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:17.394242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-ab6decb6-710f-4787-9deb-c4bb1cc6381b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:17.394242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:17.394242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:17.394242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:17.394242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:17.394242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:17.394242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:17.394242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:17.394242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:17.394242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:17.394242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:17.394242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:17.394242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:17.394242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:17.394242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:17.394242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:17.396097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:17.396097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:17.396097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:17.396097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:17.396097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:17.396097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:17.396097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:17.396097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:17.396097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:17.396097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:17.396097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:17.396097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:17.396097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:17.396097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:17.396097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:17.396097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:17.397693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:17.397693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:17.397693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:17.397693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:17.397693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:17.397693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:17.397693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:17.397693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:17.397693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:17.397693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:17.397693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:17.397693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:17.397693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 196/384] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 16 00:44:17 2022] GET /identity/v3/roles/service => generated 85 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:17.399076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7beb0770-9771-417b-b1b4-a7dee7c65c2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:17.399249 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6decb6-710f-4787-9deb-c4bb1cc6381b None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:17.399376 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7beb0770-9771-417b-b1b4-a7dee7c65c2f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:17.399376 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7beb0770-9771-417b-b1b4-a7dee7c65c2f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:17.399376 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7beb0770-9771-417b-b1b4-a7dee7c65c2f None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:17.399376 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7beb0770-9771-417b-b1b4-a7dee7c65c2f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:17.399376 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7beb0770-9771-417b-b1b4-a7dee7c65c2f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:17.401777 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7beb0770-9771-417b-b1b4-a7dee7c65c2f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:17.427030 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 189/385] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:17.427030 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7beb0770-9771-417b-b1b4-a7dee7c65c2f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:17.514070 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:17.560954 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6decb6-710f-4787-9deb-c4bb1cc6381b None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:17.577729 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7beb0770-9771-417b-b1b4-a7dee7c65c2f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:17.601614 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540a460e-3e2d-4523-9431-f262eed7eb67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:17.602237 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540a460e-3e2d-4523-9431-f262eed7eb67 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:17.602524 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540a460e-3e2d-4523-9431-f262eed7eb67 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:17.602843 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540a460e-3e2d-4523-9431-f262eed7eb67 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:17.603627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540a460e-3e2d-4523-9431-f262eed7eb67 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:17.603939 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540a460e-3e2d-4523-9431-f262eed7eb67 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:17.605358 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540a460e-3e2d-4523-9431-f262eed7eb67 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:17.618149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 197/386] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:17.619829 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540a460e-3e2d-4523-9431-f262eed7eb67 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:17.637306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c765abd-aeb3-4251-a562-c20341500331 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:17.638325 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c765abd-aeb3-4251-a562-c20341500331 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:17.638770 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c765abd-aeb3-4251-a562-c20341500331 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:17.639263 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c765abd-aeb3-4251-a562-c20341500331 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:17.657445 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c765abd-aeb3-4251-a562-c20341500331 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:17.660325 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c765abd-aeb3-4251-a562-c20341500331 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:17.688726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-0c765abd-aeb3-4251-a562-c20341500331 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:17.688726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:17.688726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:17.688726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:17.688726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:17.688726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:17.688726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:17.688726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:17.688726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:17.688726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:17.688726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:17.688726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:17.688726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:17.688726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:17.688726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:17.688726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:17.689923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:17.689923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:17.689923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:17.689923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:17.689923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:17.689923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:17.689923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:17.689923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:17.689923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:17.689923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:17.689923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:17.689923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:17.689923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:17.689923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:17.689923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:17.689923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:17.689923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:17.690773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:17.690773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:17.690773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:17.691736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:17.691817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 190/387] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:17.692410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c765abd-aeb3-4251-a562-c20341500331 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:17.737708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540a460e-3e2d-4523-9431-f262eed7eb67 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:17.779826 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb212573-d57d-4250-8543-0eb129029183 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:17.780744 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb212573-d57d-4250-8543-0eb129029183 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:17.781139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb212573-d57d-4250-8543-0eb129029183 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:17.781524 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb212573-d57d-4250-8543-0eb129029183 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:17.793449 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb212573-d57d-4250-8543-0eb129029183 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:17.794977 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb212573-d57d-4250-8543-0eb129029183 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:17.812975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-fb212573-d57d-4250-8543-0eb129029183 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:17.812975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:17.812975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:17.812975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:17.812975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:17.812975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:17.812975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:17.812975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:17.812975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:17.812975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:17.812975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:17.812975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:17.812975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:17.812975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:17.812975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:17.812975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:17.814730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:17.814730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:17.814730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:17.814730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:17.814730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:17.814730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:17.814730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:17.814730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:17.814730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:17.814730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:17.814730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:17.814730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:17.814730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:17.814730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:17.814730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:17.814730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:17.814730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:17.817303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:17.817303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:17.817303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:17.817303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 198/388] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:17.817303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb212573-d57d-4250-8543-0eb129029183 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:17.846493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c765abd-aeb3-4251-a562-c20341500331 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:17.851673 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:17.889280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d9fe92-c4b9-4b3c-b94c-61a33e5055da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:17.890035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d9fe92-c4b9-4b3c-b94c-61a33e5055da None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:17.890399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d9fe92-c4b9-4b3c-b94c-61a33e5055da None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:17.890764 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d9fe92-c4b9-4b3c-b94c-61a33e5055da None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:17.891664 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d9fe92-c4b9-4b3c-b94c-61a33e5055da None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:17.891983 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d9fe92-c4b9-4b3c-b94c-61a33e5055da None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:17.893644 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d9fe92-c4b9-4b3c-b94c-61a33e5055da None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:17.905412 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 191/389] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 16 00:44:17 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:17.906321 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d9fe92-c4b9-4b3c-b94c-61a33e5055da None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:17.957009 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb212573-d57d-4250-8543-0eb129029183 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:17.972125 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:17.997840 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fbc49a-4518-47f3-89a4-739b90760bf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:17.998488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fbc49a-4518-47f3-89a4-739b90760bf0 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:17.998768 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fbc49a-4518-47f3-89a4-739b90760bf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:17.999069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fbc49a-4518-47f3-89a4-739b90760bf0 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:18.008368 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fbc49a-4518-47f3-89a4-739b90760bf0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:18.010413 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fbc49a-4518-47f3-89a4-739b90760bf0 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:18.024220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-33fbc49a-4518-47f3-89a4-739b90760bf0 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 16 00:44:18.024220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:18.024220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:18.024220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:18.024220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:18.024220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:18.024220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:18.024220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:18.024220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:18.024220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:18.024220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:18.024220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:18.024220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:44:18.024220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:44:18.024220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:44:18.024220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:18.026922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:18.029186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:18.029186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:18.029186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:44:18.029186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:44:18.029186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:44:18.029186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:44:18.029186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:44:18.029186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:44:18.029186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 16 00:44:18.029186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:18.029186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 199/390] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:44:17 2022] GET /identity/v3/users/glance => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:18.029186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fbc49a-4518-47f3-89a4-739b90760bf0 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:18.034477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d9fe92-c4b9-4b3c-b94c-61a33e5055da None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:18.075704 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f5ff84-b0e1-4ee2-9043-bd44f6400fd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:18.076264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f5ff84-b0e1-4ee2-9043-bd44f6400fd8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:18.076550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f5ff84-b0e1-4ee2-9043-bd44f6400fd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:18.076826 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f5ff84-b0e1-4ee2-9043-bd44f6400fd8 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:18.085348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f5ff84-b0e1-4ee2-9043-bd44f6400fd8 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:18.086845 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f5ff84-b0e1-4ee2-9043-bd44f6400fd8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:18.099052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-48f5ff84-b0e1-4ee2-9043-bd44f6400fd8 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 16 00:44:18.099052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:18.099052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:18.099052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:18.099052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:18.099052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:18.099052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:18.099052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:18.099052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:18.099052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:18.099052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:18.099052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:18.099052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:44:18.099052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:44:18.099052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:44:18.099052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:18.100919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:18.102133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:18.102133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:18.102133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:44:18.102133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:44:18.102133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:44:18.102133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:44:18.102133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:44:18.102133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:44:18.102133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 16 00:44:18.102133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:18.102133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 192/391] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 16 00:44:17 2022] GET /identity/v3/users/neutron => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:18.102133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f5ff84-b0e1-4ee2-9043-bd44f6400fd8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:18.135235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:18.167334 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fbc49a-4518-47f3-89a4-739b90760bf0 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:18.204830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715f4859-7566-4671-a8a6-167d497f2b6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:18.205714 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715f4859-7566-4671-a8a6-167d497f2b6f None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:18.205991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715f4859-7566-4671-a8a6-167d497f2b6f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:18.206311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715f4859-7566-4671-a8a6-167d497f2b6f None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:18.206939 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715f4859-7566-4671-a8a6-167d497f2b6f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:18.207247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715f4859-7566-4671-a8a6-167d497f2b6f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:18.208657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715f4859-7566-4671-a8a6-167d497f2b6f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:18.221965 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 200/392] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:18.222843 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715f4859-7566-4671-a8a6-167d497f2b6f None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:18.240575 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f5ff84-b0e1-4ee2-9043-bd44f6400fd8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:18.257959 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:18.279360 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33fb946-b4c4-4de0-bb9d-bd61f70329fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:18.279984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33fb946-b4c4-4de0-bb9d-bd61f70329fb None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:18.280278 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33fb946-b4c4-4de0-bb9d-bd61f70329fb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:18.280537 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33fb946-b4c4-4de0-bb9d-bd61f70329fb None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:18.281200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33fb946-b4c4-4de0-bb9d-bd61f70329fb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:18.281532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33fb946-b4c4-4de0-bb9d-bd61f70329fb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:18.282942 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33fb946-b4c4-4de0-bb9d-bd61f70329fb None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:18.300078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 193/393] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:18.301829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33fb946-b4c4-4de0-bb9d-bd61f70329fb None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:18.343203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715f4859-7566-4671-a8a6-167d497f2b6f None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:18.381697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db21c108-a1de-4a0a-b3ec-a48104f7210e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:18.382369 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db21c108-a1de-4a0a-b3ec-a48104f7210e None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:18.382651 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db21c108-a1de-4a0a-b3ec-a48104f7210e None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:18.382956 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db21c108-a1de-4a0a-b3ec-a48104f7210e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:18.394067 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db21c108-a1de-4a0a-b3ec-a48104f7210e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:18.395618 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db21c108-a1de-4a0a-b3ec-a48104f7210e None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:18.413670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-db21c108-a1de-4a0a-b3ec-a48104f7210e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:18.413670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:18.413670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:18.413670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:18.413670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:18.413670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:18.413670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:18.413670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:18.413670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:18.413670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:18.413670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:18.413670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:18.413670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:18.413670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:18.413670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:18.413670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:18.414839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:18.414839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:18.414839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:18.414839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:18.414839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:18.414839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:18.414839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:18.414839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:18.414839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:18.414839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:18.414839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:18.414839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:18.414839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:18.414839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:18.414839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:18.414839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:18.414839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:18.417945 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:18.417945 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:18.417945 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:18.417945 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 201/394] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:18.417945 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db21c108-a1de-4a0a-b3ec-a48104f7210e None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:18.434301 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:18.452560 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33fb946-b4c4-4de0-bb9d-bd61f70329fb None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:18.499217 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315a5d06-00c7-443f-b03b-ef178299ef6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:18.499950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315a5d06-00c7-443f-b03b-ef178299ef6a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:18.500302 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315a5d06-00c7-443f-b03b-ef178299ef6a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:18.500705 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315a5d06-00c7-443f-b03b-ef178299ef6a None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:18.501542 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315a5d06-00c7-443f-b03b-ef178299ef6a None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:18.502013 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315a5d06-00c7-443f-b03b-ef178299ef6a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:18.504621 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315a5d06-00c7-443f-b03b-ef178299ef6a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:18.532796 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 194/395] 10.222.0.25 () {56 vars in 1137 bytes} [Sun Oct 16 00:44:18 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:18.533856 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315a5d06-00c7-443f-b03b-ef178299ef6a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:18.537120 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:18.560593 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db21c108-a1de-4a0a-b3ec-a48104f7210e None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:18.617722 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bef0c36-7c33-44f4-8580-92b6c9d2f3d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:18.618249 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bef0c36-7c33-44f4-8580-92b6c9d2f3d3 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:18.619094 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bef0c36-7c33-44f4-8580-92b6c9d2f3d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:18.619449 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bef0c36-7c33-44f4-8580-92b6c9d2f3d3 None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:18.620449 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bef0c36-7c33-44f4-8580-92b6c9d2f3d3 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:18.620801 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bef0c36-7c33-44f4-8580-92b6c9d2f3d3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:18.622226 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bef0c36-7c33-44f4-8580-92b6c9d2f3d3 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:18.661616 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315a5d06-00c7-443f-b03b-ef178299ef6a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:18.677868 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 202/396] 10.222.0.25 () {56 vars in 1139 bytes} [Sun Oct 16 00:44:18 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:18.678735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bef0c36-7c33-44f4-8580-92b6c9d2f3d3 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:18.720854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b75777-aa5f-414a-ad40-fbd3af8b5615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:18.721686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b75777-aa5f-414a-ad40-fbd3af8b5615 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:18.722058 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b75777-aa5f-414a-ad40-fbd3af8b5615 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:18.722435 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b75777-aa5f-414a-ad40-fbd3af8b5615 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:18.731587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b75777-aa5f-414a-ad40-fbd3af8b5615 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:18.733142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b75777-aa5f-414a-ad40-fbd3af8b5615 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:18.747371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-e3b75777-aa5f-414a-ad40-fbd3af8b5615 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:18.747371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:18.747371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:18.747371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:18.747371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:18.747371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:18.747371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:18.747371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:18.747371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:18.747371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:18.747371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:18.747371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:18.747371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:44:18.747371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:44:18.747371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:44:18.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:44:18.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:18.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:18.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:18.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:18.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:18.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:18.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:18.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:18.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:18.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:18.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:18.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:18.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:18.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:18.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:18.748954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:18.750631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:44:18.750631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:44:18.750631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:44:18.750631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:44:18.750631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:44:18.750631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:44:18.750631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:18.750631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:18.750631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 195/397] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:18.750631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b75777-aa5f-414a-ad40-fbd3af8b5615 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:18.753772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:18.842535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bef0c36-7c33-44f4-8580-92b6c9d2f3d3 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:18.896370 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b75777-aa5f-414a-ad40-fbd3af8b5615 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:18.898534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d00e6b-07bd-44ff-8355-74c6c82e0b97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:18.899281 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d00e6b-07bd-44ff-8355-74c6c82e0b97 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:18.899971 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d00e6b-07bd-44ff-8355-74c6c82e0b97 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:18.900200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d00e6b-07bd-44ff-8355-74c6c82e0b97 None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:18.901401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d00e6b-07bd-44ff-8355-74c6c82e0b97 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:18.903265 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d00e6b-07bd-44ff-8355-74c6c82e0b97 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:18.924490 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:18.935686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 203/398] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:44:18 2022] POST /identity/v3/users => generated 247 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:44:18.937706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d00e6b-07bd-44ff-8355-74c6c82e0b97 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:18.950676 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c525b6-8df2-4b44-af6b-1c9fa95b8217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:18.951358 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c525b6-8df2-4b44-af6b-1c9fa95b8217 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:18.951642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c525b6-8df2-4b44-af6b-1c9fa95b8217 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:18.951919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c525b6-8df2-4b44-af6b-1c9fa95b8217 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:18.952622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c525b6-8df2-4b44-af6b-1c9fa95b8217 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:18.952933 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c525b6-8df2-4b44-af6b-1c9fa95b8217 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:18.954545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c525b6-8df2-4b44-af6b-1c9fa95b8217 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:18.979899 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 196/399] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:18.979899 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c525b6-8df2-4b44-af6b-1c9fa95b8217 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:19.094106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:19.154153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c525b6-8df2-4b44-af6b-1c9fa95b8217 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:19.155047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d00e6b-07bd-44ff-8355-74c6c82e0b97 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:19.208618 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e27cca-0c46-4b55-93c5-1911b04291cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:19.209808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e27cca-0c46-4b55-93c5-1911b04291cc None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:19.210222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e27cca-0c46-4b55-93c5-1911b04291cc None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:19.210658 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e27cca-0c46-4b55-93c5-1911b04291cc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:19.214149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7011d4ac-7c38-4b25-9e5b-b7db877ff4f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:19.215222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7011d4ac-7c38-4b25-9e5b-b7db877ff4f3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:19.215743 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7011d4ac-7c38-4b25-9e5b-b7db877ff4f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:19.216100 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7011d4ac-7c38-4b25-9e5b-b7db877ff4f3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:19.225316 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e27cca-0c46-4b55-93c5-1911b04291cc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:19.227081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e27cca-0c46-4b55-93c5-1911b04291cc None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:19.237772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7011d4ac-7c38-4b25-9e5b-b7db877ff4f3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:19.240854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7011d4ac-7c38-4b25-9e5b-b7db877ff4f3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:19.243587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-03e27cca-0c46-4b55-93c5-1911b04291cc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:19.243587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:19.243587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:19.243587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:19.243587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:19.243587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:19.243587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:19.243587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:19.243587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:19.243587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:19.243587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:19.243587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:19.243587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:19.243587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:19.243587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:19.243587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:19.245267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:19.245267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:19.245267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:19.245267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:19.245267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:19.245267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:19.245267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:19.245267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:19.245267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:19.245267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:19.245267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:19.245267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:19.245267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:19.245267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:19.245267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:19.245267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:19.245267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:19.246311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:19.246311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:19.246311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:19.246311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 204/400] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:19.246546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e27cca-0c46-4b55-93c5-1911b04291cc None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:19.266003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-7011d4ac-7c38-4b25-9e5b-b7db877ff4f3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:19.266003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:19.266003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:19.266003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:19.266003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:19.266003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:19.266003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:19.266003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:19.266003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:19.266003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:19.266003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:19.266003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:19.266003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:19.266003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:19.266003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:19.266003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:19.268105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:19.268105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:19.268105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:19.268105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:19.268105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:19.268105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:19.268105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:19.268105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:19.268105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:19.268105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:19.268105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:19.268105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:19.268105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:19.268105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:19.268105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:19.268105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:19.268105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:19.269746 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:19.269746 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:19.269746 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:19.271863 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 197/401] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:19.273791 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7011d4ac-7c38-4b25-9e5b-b7db877ff4f3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:19.346336 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:19.438665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7011d4ac-7c38-4b25-9e5b-b7db877ff4f3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:19.453406 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e27cca-0c46-4b55-93c5-1911b04291cc None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:19.466358 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:19.499745 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caeacad1-c803-4850-abde-64fdc2553665 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:19.500672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeacad1-c803-4850-abde-64fdc2553665 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:19.501042 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeacad1-c803-4850-abde-64fdc2553665 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:19.501445 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeacad1-c803-4850-abde-64fdc2553665 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:19.512433 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caeacad1-c803-4850-abde-64fdc2553665 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:19.513915 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caeacad1-c803-4850-abde-64fdc2553665 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:19.517773 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1caf944-2b18-48a2-b8f0-073c949a81bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:19.518812 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1caf944-2b18-48a2-b8f0-073c949a81bc None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:19.519347 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1caf944-2b18-48a2-b8f0-073c949a81bc None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:19.519813 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1caf944-2b18-48a2-b8f0-073c949a81bc None admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:19.521183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1caf944-2b18-48a2-b8f0-073c949a81bc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:19.521668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1caf944-2b18-48a2-b8f0-073c949a81bc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:19.524486 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1caf944-2b18-48a2-b8f0-073c949a81bc None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:19.526642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-caeacad1-c803-4850-abde-64fdc2553665 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 16 00:44:19.526642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:19.526642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:19.526642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:19.526642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:19.526642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:19.526642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:19.526642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:19.526642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:19.526642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:19.526642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:19.526642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:19.526642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:44:19.526642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:44:19.526642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:44:19.526642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:19.528520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:19.532264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:19.532264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:19.532264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:44:19.532264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:44:19.532264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:44:19.532264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:44:19.532264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:44:19.532264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:44:19.532264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 16 00:44:19.532264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:19.532264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 198/402] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:44:19 2022] GET /identity/v3/users/cinder => generated 84 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:19.532264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caeacad1-c803-4850-abde-64fdc2553665 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:19.565399 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 205/403] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:44:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:19.566622 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1caf944-2b18-48a2-b8f0-073c949a81bc None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:19.673740 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caeacad1-c803-4850-abde-64fdc2553665 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:19.717126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36d167c-2c65-4478-9aee-0bf7bd9a73fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:19.717646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36d167c-2c65-4478-9aee-0bf7bd9a73fb None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:19.717928 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36d167c-2c65-4478-9aee-0bf7bd9a73fb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:19.718204 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36d167c-2c65-4478-9aee-0bf7bd9a73fb None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:19.718886 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36d167c-2c65-4478-9aee-0bf7bd9a73fb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:19.719217 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36d167c-2c65-4478-9aee-0bf7bd9a73fb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:19.720844 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36d167c-2c65-4478-9aee-0bf7bd9a73fb None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:19.746978 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 199/404] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:19.746978 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36d167c-2c65-4478-9aee-0bf7bd9a73fb None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:19.796248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:19.796497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1caf944-2b18-48a2-b8f0-073c949a81bc None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:19.818722 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:19.858009 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045535f3-f648-45a6-81e2-691a99f7aa58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:19.858901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045535f3-f648-45a6-81e2-691a99f7aa58 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:19.859346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045535f3-f648-45a6-81e2-691a99f7aa58 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:19.859757 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045535f3-f648-45a6-81e2-691a99f7aa58 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:19.860594 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045535f3-f648-45a6-81e2-691a99f7aa58 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:19.861033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045535f3-f648-45a6-81e2-691a99f7aa58 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:19.862637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045535f3-f648-45a6-81e2-691a99f7aa58 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:19.886640 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 206/405] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:19.886640 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045535f3-f648-45a6-81e2-691a99f7aa58 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:19.924709 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36d167c-2c65-4478-9aee-0bf7bd9a73fb None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:19.982901 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d039dee-85b8-4980-a032-7053a46e34e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:19.983788 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d039dee-85b8-4980-a032-7053a46e34e1 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:19.984360 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d039dee-85b8-4980-a032-7053a46e34e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:19.984515 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d039dee-85b8-4980-a032-7053a46e34e1 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:19.986211 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d039dee-85b8-4980-a032-7053a46e34e1 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:19.986211 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d039dee-85b8-4980-a032-7053a46e34e1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:19.988667 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d039dee-85b8-4980-a032-7053a46e34e1 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:20.018997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 200/406] 10.222.0.25 () {56 vars in 1137 bytes} [Sun Oct 16 00:44:19 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:20.020873 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d039dee-85b8-4980-a032-7053a46e34e1 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:20.059359 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045535f3-f648-45a6-81e2-691a99f7aa58 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:20.120555 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777ff60c-c6bc-403e-b7b8-f38e8206fa8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:20.121530 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777ff60c-c6bc-403e-b7b8-f38e8206fa8d None admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:20.121994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777ff60c-c6bc-403e-b7b8-f38e8206fa8d None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:20.122444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777ff60c-c6bc-403e-b7b8-f38e8206fa8d None admin] PATH_INFO: `/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/4eba26813e154595a12a67fe4be5eeff/roles/bfe57c41a4da4b25b9b3e4761f220a46` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:20.149095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777ff60c-c6bc-403e-b7b8-f38e8206fa8d None admin] RBAC: Authorizing `identity:create_grant(project_id=7deb2c3dae214df7895b40e7339bd6ea, user_id=4eba26813e154595a12a67fe4be5eeff, role_id=bfe57c41a4da4b25b9b3e4761f220a46)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:20.151778 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777ff60c-c6bc-403e-b7b8-f38e8206fa8d None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:20.174284 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 207/407] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:44:19 2022] PUT /identity/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/4eba26813e154595a12a67fe4be5eeff/roles/bfe57c41a4da4b25b9b3e4761f220a46 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:44:20.176555 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777ff60c-c6bc-403e-b7b8-f38e8206fa8d None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:20.196149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:20.212770 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d039dee-85b8-4980-a032-7053a46e34e1 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:20.235079 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:20.285581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc17bbb-9ce0-4848-abd6-f72dd562cc1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:20.287407 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc17bbb-9ce0-4848-abd6-f72dd562cc1b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:20.288138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc17bbb-9ce0-4848-abd6-f72dd562cc1b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:20.288876 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc17bbb-9ce0-4848-abd6-f72dd562cc1b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:20.302328 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc17bbb-9ce0-4848-abd6-f72dd562cc1b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:20.305962 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc17bbb-9ce0-4848-abd6-f72dd562cc1b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:20.329522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-6fc17bbb-9ce0-4848-abd6-f72dd562cc1b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:20.329522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:20.329522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:20.329522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:20.329522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:20.329522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:20.329522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:20.329522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:20.329522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:20.329522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:20.329522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:20.329522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:20.329522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:44:20.329522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:44:20.329522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:44:20.331401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:44:20.331401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:20.331401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:20.331401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:20.331401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:20.331401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:20.331401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:20.331401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:20.331401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:20.331401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:20.331401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:20.331401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:20.331401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:20.331401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:20.331401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:20.331401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:20.331401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:20.333002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:44:20.333002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:44:20.333002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:44:20.333002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:44:20.333002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:44:20.333002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:44:20.333002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:20.333002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:20.336721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 201/408] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:20.339153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc17bbb-9ce0-4848-abd6-f72dd562cc1b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:20.418378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777ff60c-c6bc-403e-b7b8-f38e8206fa8d None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:20.487987 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff527c8-4c58-492b-ba55-c9b6dc16804f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:20.488960 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff527c8-4c58-492b-ba55-c9b6dc16804f None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:20.489441 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff527c8-4c58-492b-ba55-c9b6dc16804f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:20.489926 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff527c8-4c58-492b-ba55-c9b6dc16804f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:20.504099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff527c8-4c58-492b-ba55-c9b6dc16804f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:20.506935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff527c8-4c58-492b-ba55-c9b6dc16804f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:20.534601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-1ff527c8-4c58-492b-ba55-c9b6dc16804f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:20.534601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:20.534601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:20.534601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:20.534601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:20.534601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:20.534601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:20.534601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:20.534601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:20.534601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:20.534601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:20.534601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:20.534601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:44:20.534601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:44:20.534601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:44:20.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:44:20.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:20.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:20.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:20.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:20.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:20.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:20.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:20.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:20.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:20.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:20.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:20.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:20.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:20.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:20.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:20.536157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:20.537788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:44:20.537788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:44:20.537788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:44:20.537788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:44:20.537788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:44:20.537788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:44:20.537788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:20.537788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:20.537788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 208/409] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 362 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:20.547517 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff527c8-4c58-492b-ba55-c9b6dc16804f None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:20.592555 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc17bbb-9ce0-4848-abd6-f72dd562cc1b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:20.682643 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b98bb36-0b3c-4a74-a9ff-54e9a9b1b981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:20.684188 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b98bb36-0b3c-4a74-a9ff-54e9a9b1b981 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:20.684952 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b98bb36-0b3c-4a74-a9ff-54e9a9b1b981 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:20.685715 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b98bb36-0b3c-4a74-a9ff-54e9a9b1b981 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:20.708737 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b98bb36-0b3c-4a74-a9ff-54e9a9b1b981 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:20.712798 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b98bb36-0b3c-4a74-a9ff-54e9a9b1b981 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:20.727066 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:20.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-9b98bb36-0b3c-4a74-a9ff-54e9a9b1b981 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:20.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:20.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:20.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:20.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:20.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:20.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:20.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:20.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:20.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:20.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:20.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:20.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:20.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:20.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:20.741930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:20.744104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:20.744104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:20.744104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:20.744104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:20.744104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:20.744104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:20.744104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:20.744104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:20.744104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:20.744104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:20.744104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:20.744104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:20.744104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:20.744104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:20.744104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:20.744104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:20.744104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:20.745797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:20.745797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:20.745797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:20.749543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 202/410] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 412 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:20.751695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b98bb36-0b3c-4a74-a9ff-54e9a9b1b981 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:20.769954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:20.771797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff527c8-4c58-492b-ba55-c9b6dc16804f None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:20.846212 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df534b40-5aeb-471e-b7ca-d5b5851655fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:20.847317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df534b40-5aeb-471e-b7ca-d5b5851655fa None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:20.847821 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df534b40-5aeb-471e-b7ca-d5b5851655fa None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:20.848281 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df534b40-5aeb-471e-b7ca-d5b5851655fa None admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:20.849583 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df534b40-5aeb-471e-b7ca-d5b5851655fa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:20.850136 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df534b40-5aeb-471e-b7ca-d5b5851655fa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:20.853997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df534b40-5aeb-471e-b7ca-d5b5851655fa None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:20.886898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 209/411] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:44:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 344 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:20.886898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df534b40-5aeb-471e-b7ca-d5b5851655fa None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:20.977342 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b98bb36-0b3c-4a74-a9ff-54e9a9b1b981 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:21.039214 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59a2728-90f3-492c-8857-45a548a4cffa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:21.040081 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59a2728-90f3-492c-8857-45a548a4cffa None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:21.040597 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59a2728-90f3-492c-8857-45a548a4cffa None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:21.041043 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59a2728-90f3-492c-8857-45a548a4cffa None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:21.042226 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59a2728-90f3-492c-8857-45a548a4cffa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:21.042729 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59a2728-90f3-492c-8857-45a548a4cffa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:21.045483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59a2728-90f3-492c-8857-45a548a4cffa None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:21.078410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 203/412] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:44:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:21.079674 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59a2728-90f3-492c-8857-45a548a4cffa None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:21.096496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df534b40-5aeb-471e-b7ca-d5b5851655fa None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:21.162397 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c473adc2-8ce0-4b9b-aa29-c54cffb236cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:21.163502 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c473adc2-8ce0-4b9b-aa29-c54cffb236cf None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:21.164031 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c473adc2-8ce0-4b9b-aa29-c54cffb236cf None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:21.164958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c473adc2-8ce0-4b9b-aa29-c54cffb236cf None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:21.166808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c473adc2-8ce0-4b9b-aa29-c54cffb236cf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:21.167513 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c473adc2-8ce0-4b9b-aa29-c54cffb236cf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:21.170235 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c473adc2-8ce0-4b9b-aa29-c54cffb236cf None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:21.186773 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:21.198773 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 210/413] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:21.200446 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c473adc2-8ce0-4b9b-aa29-c54cffb236cf None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:21.221290 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:21.287653 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59a2728-90f3-492c-8857-45a548a4cffa None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:21.356682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5092c992-05bd-4063-ba87-805f27d2f0fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:21.357763 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092c992-05bd-4063-ba87-805f27d2f0fb None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:21.358234 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092c992-05bd-4063-ba87-805f27d2f0fb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:21.358696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092c992-05bd-4063-ba87-805f27d2f0fb None admin] PATH_INFO: `/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/78473c29cef449b89bc9185de67f1bf7/roles/bfe57c41a4da4b25b9b3e4761f220a46` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:21.386706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5092c992-05bd-4063-ba87-805f27d2f0fb None admin] RBAC: Authorizing `identity:create_grant(project_id=7deb2c3dae214df7895b40e7339bd6ea, user_id=78473c29cef449b89bc9185de67f1bf7, role_id=bfe57c41a4da4b25b9b3e4761f220a46)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:21.389025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5092c992-05bd-4063-ba87-805f27d2f0fb None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:21.414777 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 204/414] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:44:21 2022] PUT /identity/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/78473c29cef449b89bc9185de67f1bf7/roles/bfe57c41a4da4b25b9b3e4761f220a46 => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:44:21.417158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5092c992-05bd-4063-ba87-805f27d2f0fb None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:21.424490 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c473adc2-8ce0-4b9b-aa29-c54cffb236cf None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:21.495439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35614dc-5e92-4894-9655-7935358d88d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:21.497079 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35614dc-5e92-4894-9655-7935358d88d8 None admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:21.497830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35614dc-5e92-4894-9655-7935358d88d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:21.498662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35614dc-5e92-4894-9655-7935358d88d8 None admin] PATH_INFO: `/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/f4eb1a640186411097cb1e3bb9e92020/roles/bfe57c41a4da4b25b9b3e4761f220a46` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:21.528883 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35614dc-5e92-4894-9655-7935358d88d8 None admin] RBAC: Authorizing `identity:create_grant(project_id=7deb2c3dae214df7895b40e7339bd6ea, user_id=f4eb1a640186411097cb1e3bb9e92020, role_id=bfe57c41a4da4b25b9b3e4761f220a46)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:21.532016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35614dc-5e92-4894-9655-7935358d88d8 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:21.585107 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 211/415] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:44:21 2022] PUT /identity/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/f4eb1a640186411097cb1e3bb9e92020/roles/bfe57c41a4da4b25b9b3e4761f220a46 => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:44:21.669336 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5092c992-05bd-4063-ba87-805f27d2f0fb None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:21.709097 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:21.742716 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2ea0b2-587a-416a-a2bf-aaa4ab74f3b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:21.743890 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2ea0b2-587a-416a-a2bf-aaa4ab74f3b7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:21.744560 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2ea0b2-587a-416a-a2bf-aaa4ab74f3b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:21.745055 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2ea0b2-587a-416a-a2bf-aaa4ab74f3b7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:21.759969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2ea0b2-587a-416a-a2bf-aaa4ab74f3b7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:21.763330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2ea0b2-587a-416a-a2bf-aaa4ab74f3b7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:21.789173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-bb2ea0b2-587a-416a-a2bf-aaa4ab74f3b7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:21.789173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:21.789173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:21.789173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:21.789173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:21.789173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:21.789173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:21.789173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:21.789173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:21.789173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:21.789173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:21.789173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:21.789173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:44:21.789173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:44:21.789173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:44:21.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:44:21.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:21.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:21.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:21.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:21.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:21.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:21.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:21.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:21.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:21.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:21.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:21.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:21.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:21.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:21.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:21.790262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:21.791369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:44:21.791369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:44:21.791369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:44:21.791369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:44:21.791369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:44:21.791369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:44:21.791369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:21.791369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:21.792295 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 205/416] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 377 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:21.801897 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35614dc-5e92-4894-9655-7935358d88d8 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:21.878939 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:21.955680 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35614dc-5e92-4894-9655-7935358d88d8 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:22.011740 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32482bb6-c7fd-4bad-ab7d-69fcdde451ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:22.012376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32482bb6-c7fd-4bad-ab7d-69fcdde451ea None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:22.012655 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32482bb6-c7fd-4bad-ab7d-69fcdde451ea None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:22.012933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32482bb6-c7fd-4bad-ab7d-69fcdde451ea None admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:22.013642 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32482bb6-c7fd-4bad-ab7d-69fcdde451ea None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:22.013949 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32482bb6-c7fd-4bad-ab7d-69fcdde451ea None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:22.015444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32482bb6-c7fd-4bad-ab7d-69fcdde451ea None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:22.029502 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 212/417] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:44:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:22.035333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2ea0b2-587a-416a-a2bf-aaa4ab74f3b7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:22.170275 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2ea0b2-587a-416a-a2bf-aaa4ab74f3b7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:22.199015 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c9d804-9c95-40e0-9d9c-fd2573de633e None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:22.199424 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c9d804-9c95-40e0-9d9c-fd2573de633e None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:22.199596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c9d804-9c95-40e0-9d9c-fd2573de633e None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:22.200940 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 213/418] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:44:22.210182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28d67e3-dfa6-4df4-b1c5-781af52977b2 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:22.210402 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28d67e3-dfa6-4df4-b1c5-781af52977b2 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:22.210697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28d67e3-dfa6-4df4-b1c5-781af52977b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:22.248343 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66446855-cbff-4fbc-9dd9-30750b7a60b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:22.249125 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66446855-cbff-4fbc-9dd9-30750b7a60b5 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:22.249440 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66446855-cbff-4fbc-9dd9-30750b7a60b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:22.249586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66446855-cbff-4fbc-9dd9-30750b7a60b5 None admin] PATH_INFO: `/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/81fa2042309c48e1859b601d8bee1d5b/roles/bfe57c41a4da4b25b9b3e4761f220a46` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:22.251456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-c28d67e3-dfa6-4df4-b1c5-781af52977b2 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:22.252713 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c28d67e3-dfa6-4df4-b1c5-781af52977b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:22.280482 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66446855-cbff-4fbc-9dd9-30750b7a60b5 None admin] RBAC: Authorizing `identity:create_grant(project_id=7deb2c3dae214df7895b40e7339bd6ea, user_id=81fa2042309c48e1859b601d8bee1d5b, role_id=bfe57c41a4da4b25b9b3e4761f220a46)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:22.286918 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66446855-cbff-4fbc-9dd9-30750b7a60b5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:22.300816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 206/419] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:44:22 2022] PUT /identity/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/81fa2042309c48e1859b601d8bee1d5b/roles/bfe57c41a4da4b25b9b3e4761f220a46 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:44:22.393876 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 214/420] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:44:22.705931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66446855-cbff-4fbc-9dd9-30750b7a60b5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:22.738674 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:22.845950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66446855-cbff-4fbc-9dd9-30750b7a60b5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:22.896621 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6956f7c9-d0b4-4d01-8691-9608588bc07b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:22.897722 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6956f7c9-d0b4-4d01-8691-9608588bc07b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:22.898205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6956f7c9-d0b4-4d01-8691-9608588bc07b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:22.898673 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6956f7c9-d0b4-4d01-8691-9608588bc07b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:22.918480 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6956f7c9-d0b4-4d01-8691-9608588bc07b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:22.923895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-6956f7c9-d0b4-4d01-8691-9608588bc07b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:22.923895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:22.923895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:22.923895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:22.923895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:22.923895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:22.923895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:22.923895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:22.923895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:22.923895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:22.923895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:22.923895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:22.923895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:22.923895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:22.923895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:22.923895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:22.924820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:22.924820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:22.924820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:22.924820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:22.924820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:22.924820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:22.924820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:22.924820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:22.924820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:22.924820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:22.924820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:22.924820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:22.924820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:22.924820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:22.924820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:22.924820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:22.925650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:22.925650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:22.925650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:22.925650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:22.925650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:22.925650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:22.925650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:22.925650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:22.925650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:22.925650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:22.925650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:22.925650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:22.928092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 207/421] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 16 00:44:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:22.935544 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28d67e3-dfa6-4df4-b1c5-781af52977b2 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:23.021548 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:23.113236 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28d67e3-dfa6-4df4-b1c5-781af52977b2 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:23.164992 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ac589b-aa1d-414b-af39-650a779986e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:23.165684 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ac589b-aa1d-414b-af39-650a779986e5 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:23.165966 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ac589b-aa1d-414b-af39-650a779986e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:23.166232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ac589b-aa1d-414b-af39-650a779986e5 None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:23.166934 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ac589b-aa1d-414b-af39-650a779986e5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:23.167255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ac589b-aa1d-414b-af39-650a779986e5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:23.179508 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ac589b-aa1d-414b-af39-650a779986e5 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:23.195627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 215/422] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 16 00:44:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:23.202244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6956f7c9-d0b4-4d01-8691-9608588bc07b None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:23.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:23.390571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6956f7c9-d0b4-4d01-8691-9608588bc07b None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:23.469112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1641a44c-27c1-4ea2-9963-7523223aaed9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:23.469857 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1641a44c-27c1-4ea2-9963-7523223aaed9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:23.470234 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1641a44c-27c1-4ea2-9963-7523223aaed9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:23.470510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1641a44c-27c1-4ea2-9963-7523223aaed9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:23.471303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69f2af5-497f-4c8c-a6c9-b5444d61a663 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:23.471787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69f2af5-497f-4c8c-a6c9-b5444d61a663 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:23.472421 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69f2af5-497f-4c8c-a6c9-b5444d61a663 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:23.474820 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 216/423] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:23 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:44:23.481919 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7b16cd-c834-4cc7-a1f3-5202404bcba9 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:23.482201 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7b16cd-c834-4cc7-a1f3-5202404bcba9 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:23.482480 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7b16cd-c834-4cc7-a1f3-5202404bcba9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:23.486957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1641a44c-27c1-4ea2-9963-7523223aaed9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:23.488920 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1641a44c-27c1-4ea2-9963-7523223aaed9 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:23.507947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-1641a44c-27c1-4ea2-9963-7523223aaed9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:23.507947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:23.507947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:23.507947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:23.507947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:23.507947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:23.507947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:23.507947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:23.507947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:23.507947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:23.507947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:23.507947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:23.507947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:23.507947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:23.507947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:23.507947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:23.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:23.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:23.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:23.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:23.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:23.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:23.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:23.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:23.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:23.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:23.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:23.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:23.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:23.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:23.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:23.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:23.509950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:23.511361 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:23.511361 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:23.511361 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:23.513813 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 208/424] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:23.519718 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1641a44c-27c1-4ea2-9963-7523223aaed9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:23.527462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-7c7b16cd-c834-4cc7-a1f3-5202404bcba9 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:23.529161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-7c7b16cd-c834-4cc7-a1f3-5202404bcba9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:23.683870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 217/425] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:23.751111 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1641a44c-27c1-4ea2-9963-7523223aaed9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:23.826972 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c89828c-b3e8-4ea5-91a4-cda9444d56e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:23.826972 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c89828c-b3e8-4ea5-91a4-cda9444d56e8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:23.827281 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c89828c-b3e8-4ea5-91a4-cda9444d56e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:23.827607 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c89828c-b3e8-4ea5-91a4-cda9444d56e8 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:23.829825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c89828c-b3e8-4ea5-91a4-cda9444d56e8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:23.831042 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c89828c-b3e8-4ea5-91a4-cda9444d56e8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:23.833038 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c89828c-b3e8-4ea5-91a4-cda9444d56e8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:23.849451 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:23.860446 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 209/426] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 341 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:23.865730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7b16cd-c834-4cc7-a1f3-5202404bcba9 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:23.936616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:24.074028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7b16cd-c834-4cc7-a1f3-5202404bcba9 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:24.074345 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c89828c-b3e8-4ea5-91a4-cda9444d56e8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:24.142138 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f0e4f4-c38b-4133-be42-5e19ba83ec5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:24.143279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f0e4f4-c38b-4133-be42-5e19ba83ec5f None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:24.144233 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f0e4f4-c38b-4133-be42-5e19ba83ec5f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:24.144745 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f0e4f4-c38b-4133-be42-5e19ba83ec5f None admin] PATH_INFO: `/v3/users/swift` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:24.160103 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f0e4f4-c38b-4133-be42-5e19ba83ec5f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:24.163829 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f0e4f4-c38b-4133-be42-5e19ba83ec5f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:24.194182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-c8f0e4f4-c38b-4133-be42-5e19ba83ec5f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 16 00:44:24.194182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:24.194182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:24.194182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:24.194182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:24.194182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:24.194182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:24.194182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:24.194182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:24.194182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:24.194182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:24.194182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:24.194182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:44:24.194182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:44:24.194182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:44:24.194182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:24.195139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:24.196045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:24.196045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:24.196045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:44:24.196045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:44:24.196045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:44:24.196045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:44:24.196045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:44:24.196045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:44:24.196045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 16 00:44:24.196045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:24.199274 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 218/427] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:44:23 2022] GET /identity/v3/users/swift => generated 83 bytes in 330 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:24.206702 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f0e4f4-c38b-4133-be42-5e19ba83ec5f None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:24.325577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c89828c-b3e8-4ea5-91a4-cda9444d56e8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:24.416207 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f0e4f4-c38b-4133-be42-5e19ba83ec5f None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:24.419289 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:24.471756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6825ec-4d21-416d-9b3a-1d4218330780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:24.472375 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6825ec-4d21-416d-9b3a-1d4218330780 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:24.472590 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6825ec-4d21-416d-9b3a-1d4218330780 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:24.472846 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6825ec-4d21-416d-9b3a-1d4218330780 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:24.480737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b061e17e-7d34-40fa-872d-2470aff9637c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:24.481812 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b061e17e-7d34-40fa-872d-2470aff9637c None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:24.482419 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b061e17e-7d34-40fa-872d-2470aff9637c None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:24.483132 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b061e17e-7d34-40fa-872d-2470aff9637c None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:24.485349 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b061e17e-7d34-40fa-872d-2470aff9637c None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:24.485899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b061e17e-7d34-40fa-872d-2470aff9637c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:24.487654 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b061e17e-7d34-40fa-872d-2470aff9637c None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:24.507889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6825ec-4d21-416d-9b3a-1d4218330780 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:24.513744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-2e6825ec-4d21-416d-9b3a-1d4218330780 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:24.513744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:24.513744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:24.513744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:24.513744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:24.513744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:24.513744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:24.513744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:24.513744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:24.513744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:24.513744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:24.513744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:24.513744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:24.513744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:24.513744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:24.513744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:24.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:24.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:24.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:24.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:24.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:24.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:24.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:24.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:24.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:24.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:24.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:24.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:24.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:24.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:24.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:24.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:24.515875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:24.515875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:24.515875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:24.515875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:24.515875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:24.515875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:24.515875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:24.515875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:24.515875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:24.515875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:24.515875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:24.515875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:24.517522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 210/428] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 16 00:44:24 2022] GET /identity/v3/roles/service => generated 85 bytes in 444 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:24.526399 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 219/429] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Oct 16 00:44:24 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:24.527261 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6825ec-4d21-416d-9b3a-1d4218330780 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:24.533004 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b061e17e-7d34-40fa-872d-2470aff9637c None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:24.681723 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:24.713994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b061e17e-7d34-40fa-872d-2470aff9637c None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:24.751496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6825ec-4d21-416d-9b3a-1d4218330780 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:24.753404 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f1f04b-a6db-4b1a-aac3-fb070bb78ea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:24.754230 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f1f04b-a6db-4b1a-aac3-fb070bb78ea9 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:24.754632 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f1f04b-a6db-4b1a-aac3-fb070bb78ea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:24.755169 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f1f04b-a6db-4b1a-aac3-fb070bb78ea9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:24.768059 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f1f04b-a6db-4b1a-aac3-fb070bb78ea9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:24.769681 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f1f04b-a6db-4b1a-aac3-fb070bb78ea9 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:24.787488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-80f1f04b-a6db-4b1a-aac3-fb070bb78ea9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:24.787488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:24.787488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:24.787488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:24.787488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:24.787488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:24.787488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:24.787488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:24.787488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:24.787488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:24.787488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:24.787488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:24.787488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:24.787488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:24.787488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:24.787488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:24.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:24.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:24.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:24.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:24.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:24.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:24.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:24.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:24.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:24.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:24.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:24.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:24.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:24.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:24.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:24.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:24.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:24.790521 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:24.790521 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:24.790521 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:24.790521 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 220/430] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:24.795820 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f157dda9-c842-496a-bc5a-e650eeeebc51 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:24.796159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f157dda9-c842-496a-bc5a-e650eeeebc51 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:24.796507 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f157dda9-c842-496a-bc5a-e650eeeebc51 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:24.798332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 221/431] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:24 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:24.799388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f157dda9-c842-496a-bc5a-e650eeeebc51 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:24.802915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:24.817811 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84ef88e-22d5-4a0c-b0ce-682e708d83db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:24.819168 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84ef88e-22d5-4a0c-b0ce-682e708d83db None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:24.820067 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84ef88e-22d5-4a0c-b0ce-682e708d83db None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:24.820792 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84ef88e-22d5-4a0c-b0ce-682e708d83db None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:24.822856 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84ef88e-22d5-4a0c-b0ce-682e708d83db None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:24.823838 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84ef88e-22d5-4a0c-b0ce-682e708d83db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:24.826999 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84ef88e-22d5-4a0c-b0ce-682e708d83db None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:24.840013 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 211/432] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 16 00:44:24 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:24.842692 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21b2ba6-fe48-4792-b1be-7b06832a5ede None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:24.843322 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21b2ba6-fe48-4792-b1be-7b06832a5ede None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:24.844140 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21b2ba6-fe48-4792-b1be-7b06832a5ede None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:24.869632 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b21b2ba6-fe48-4792-b1be-7b06832a5ede None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:24.871677 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b21b2ba6-fe48-4792-b1be-7b06832a5ede None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:24.924651 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f157dda9-c842-496a-bc5a-e650eeeebc51 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:24.975848 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33d96fc-6b8c-400b-aeb2-01ec01eab943 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:24.976862 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33d96fc-6b8c-400b-aeb2-01ec01eab943 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:24.977278 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33d96fc-6b8c-400b-aeb2-01ec01eab943 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:24.978379 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33d96fc-6b8c-400b-aeb2-01ec01eab943 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:24.979717 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33d96fc-6b8c-400b-aeb2-01ec01eab943 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:24.980360 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33d96fc-6b8c-400b-aeb2-01ec01eab943 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:24.982376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33d96fc-6b8c-400b-aeb2-01ec01eab943 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:25.002502 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 222/433] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:25.008235 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33d96fc-6b8c-400b-aeb2-01ec01eab943 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:25.041954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 212/434] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:25.044906 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21b2ba6-fe48-4792-b1be-7b06832a5ede None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:25.138713 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:25.179545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:25.224938 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33d96fc-6b8c-400b-aeb2-01ec01eab943 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:25.297068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21b2ba6-fe48-4792-b1be-7b06832a5ede None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:25.304707 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f65fed-332a-4daa-80b1-3466099fc8d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:25.306528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f65fed-332a-4daa-80b1-3466099fc8d7 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:25.307179 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f65fed-332a-4daa-80b1-3466099fc8d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:25.307754 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f65fed-332a-4daa-80b1-3466099fc8d7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:25.328995 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f65fed-332a-4daa-80b1-3466099fc8d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:25.332348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f65fed-332a-4daa-80b1-3466099fc8d7 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:25.363215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-49f65fed-332a-4daa-80b1-3466099fc8d7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:25.363215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:25.363215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:25.363215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:25.363215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:25.363215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:25.363215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:25.363215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:25.363215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:25.363215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:25.363215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:25.363215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:25.363215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:25.363215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:25.363215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:25.363215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:25.364227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:25.364227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:25.364227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:25.364227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:25.364227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:25.364227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:25.364227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:25.364227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:25.364227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:25.364227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:25.364227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:25.364227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:25.364227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:25.364227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:25.364227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:25.364227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:25.364227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:25.365089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:25.365089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:25.365089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:25.366175 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 223/435] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 362 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:25.367647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4850bf22-86da-4993-9056-bd95105b4f34 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:25.367883 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4850bf22-86da-4993-9056-bd95105b4f34 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:25.368746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4850bf22-86da-4993-9056-bd95105b4f34 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:25.374924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 224/436] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:25 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:25.374924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4850bf22-86da-4993-9056-bd95105b4f34 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:25.380985 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f647062-bbb4-4b85-9d37-6dbf93fe92ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:25.382214 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f647062-bbb4-4b85-9d37-6dbf93fe92ef None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:25.382709 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f647062-bbb4-4b85-9d37-6dbf93fe92ef None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:25.383217 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f647062-bbb4-4b85-9d37-6dbf93fe92ef None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:25.396158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f647062-bbb4-4b85-9d37-6dbf93fe92ef None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:25.399044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f647062-bbb4-4b85-9d37-6dbf93fe92ef None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:25.421077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-7f647062-bbb4-4b85-9d37-6dbf93fe92ef None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:25.421077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:25.421077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:25.421077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:25.421077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:25.421077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:25.421077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:25.421077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:25.421077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:25.421077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:25.421077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:25.421077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:25.421077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:44:25.421077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:44:25.421077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:44:25.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:44:25.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:25.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:25.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:25.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:25.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:25.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:25.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:25.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:25.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:25.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:25.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:25.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:25.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:25.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:25.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:25.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:25.424608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:44:25.424608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:44:25.424608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:44:25.424608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:44:25.424608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:44:25.424608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:44:25.424608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:25.424608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:25.430327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 213/437] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 387 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:25.431936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f647062-bbb4-4b85-9d37-6dbf93fe92ef None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:25.570279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4850bf22-86da-4993-9056-bd95105b4f34 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:25.590289 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:25.624092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f03c7c-d001-4109-9a3c-80bfd3647f14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:25.625169 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f03c7c-d001-4109-9a3c-80bfd3647f14 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:25.625670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f03c7c-d001-4109-9a3c-80bfd3647f14 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:25.626165 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f03c7c-d001-4109-9a3c-80bfd3647f14 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:25.627189 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f647062-bbb4-4b85-9d37-6dbf93fe92ef None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:25.643850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:25.650414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f03c7c-d001-4109-9a3c-80bfd3647f14 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:25.660482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-98f03c7c-d001-4109-9a3c-80bfd3647f14 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:25.660482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:25.660482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:25.660482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:25.660482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:25.660482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:25.660482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:25.660482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:25.660482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:25.660482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:25.660482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:25.660482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:25.660482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:25.660482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:25.660482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:25.660482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:25.662095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:25.662095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:25.662095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:25.662095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:25.662095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:25.662095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:25.662095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:25.662095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:25.662095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:25.662095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:25.662095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:25.662095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:25.662095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:25.662095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:25.662095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:25.662095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:25.664415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:25.664415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:25.664415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:25.664415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:25.664415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:25.664415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:25.664415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:25.664415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:25.664415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:25.664415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:25.664415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:25.664415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:25.664415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 225/438] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 16 00:44:25 2022] GET /identity/v3/roles/service => generated 85 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:25.667589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d69411-20a4-4bd9-98a3-ccb4f33c2d1d None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:25.667589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d69411-20a4-4bd9-98a3-ccb4f33c2d1d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:25.667589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d69411-20a4-4bd9-98a3-ccb4f33c2d1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:25.687171 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22cde101-09ba-47fb-9c04-a1ff0ae61a89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:25.688385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cde101-09ba-47fb-9c04-a1ff0ae61a89 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:25.688974 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cde101-09ba-47fb-9c04-a1ff0ae61a89 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:25.689500 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cde101-09ba-47fb-9c04-a1ff0ae61a89 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:25.690435 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22cde101-09ba-47fb-9c04-a1ff0ae61a89 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:25.691929 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22cde101-09ba-47fb-9c04-a1ff0ae61a89 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:25.694256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22cde101-09ba-47fb-9c04-a1ff0ae61a89 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:25.699422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-22d69411-20a4-4bd9-98a3-ccb4f33c2d1d None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:25.701404 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-22d69411-20a4-4bd9-98a3-ccb4f33c2d1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:25.710504 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 214/439] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:25.712007 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22cde101-09ba-47fb-9c04-a1ff0ae61a89 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:25.801850 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 226/440] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:25.803076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d69411-20a4-4bd9-98a3-ccb4f33c2d1d None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:25.896456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22cde101-09ba-47fb-9c04-a1ff0ae61a89 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:25.941066 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409cd158-f27a-419e-9769-9c222b3307aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:25.942098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409cd158-f27a-419e-9769-9c222b3307aa None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:25.942546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409cd158-f27a-419e-9769-9c222b3307aa None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:25.943016 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409cd158-f27a-419e-9769-9c222b3307aa None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:25.944210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409cd158-f27a-419e-9769-9c222b3307aa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:25.944708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409cd158-f27a-419e-9769-9c222b3307aa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:25.947439 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409cd158-f27a-419e-9769-9c222b3307aa None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:25.969394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 215/441] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:44:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:25.971583 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d69411-20a4-4bd9-98a3-ccb4f33c2d1d None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:25.972799 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409cd158-f27a-419e-9769-9c222b3307aa None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:25.977070 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:26.027010 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:26.036567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21786e7a-82f1-4f6d-a937-a40d69c9a2ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:26.037497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21786e7a-82f1-4f6d-a937-a40d69c9a2ad None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:26.037965 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21786e7a-82f1-4f6d-a937-a40d69c9a2ad None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:26.038421 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21786e7a-82f1-4f6d-a937-a40d69c9a2ad None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:26.039599 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21786e7a-82f1-4f6d-a937-a40d69c9a2ad None admin] RBAC: Adding query filter params (name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:26.040173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21786e7a-82f1-4f6d-a937-a40d69c9a2ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:26.042596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21786e7a-82f1-4f6d-a937-a40d69c9a2ad None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:26.056230 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 227/442] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 16 00:44:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:26.059165 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bfdc38-59c3-42bf-8aee-5f0a3ae0d13d None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:26.059930 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bfdc38-59c3-42bf-8aee-5f0a3ae0d13d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:26.060675 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bfdc38-59c3-42bf-8aee-5f0a3ae0d13d None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:26.068557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 228/443] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:26 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:26.068557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bfdc38-59c3-42bf-8aee-5f0a3ae0d13d None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:26.161505 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409cd158-f27a-419e-9769-9c222b3307aa None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:26.192935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bfdc38-59c3-42bf-8aee-5f0a3ae0d13d None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:26.204750 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4933ac5a-4968-4390-b8c5-1c5487abbb53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:26.206006 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933ac5a-4968-4390-b8c5-1c5487abbb53 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:26.206397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933ac5a-4968-4390-b8c5-1c5487abbb53 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:26.206782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933ac5a-4968-4390-b8c5-1c5487abbb53 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:26.216741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4933ac5a-4968-4390-b8c5-1c5487abbb53 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:26.218538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4933ac5a-4968-4390-b8c5-1c5487abbb53 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:26.232756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-4933ac5a-4968-4390-b8c5-1c5487abbb53 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 16 00:44:26.232756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:26.232756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:26.232756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:26.232756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:26.232756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:26.232756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:26.232756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:26.232756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:26.232756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:26.232756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:26.232756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:26.232756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:44:26.232756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:44:26.232756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:44:26.232756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:26.234492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:26.236350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:26.236350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:26.236350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:44:26.236350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:44:26.236350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:44:26.236350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:44:26.236350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:44:26.236350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:44:26.236350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 16 00:44:26.236350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:26.237124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf1c4cf-7a96-4f8c-a025-c9fb9792bbc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:26.237124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf1c4cf-7a96-4f8c-a025-c9fb9792bbc8 None admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:26.237124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf1c4cf-7a96-4f8c-a025-c9fb9792bbc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:26.237124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf1c4cf-7a96-4f8c-a025-c9fb9792bbc8 None admin] PATH_INFO: `/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/de64d498da78448394b0d4d1851a2789/roles/bfe57c41a4da4b25b9b3e4761f220a46` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:26.237386 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 216/444] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Oct 16 00:44:25 2022] GET /identity/v3/users/nova => generated 82 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:26.237386 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4933ac5a-4968-4390-b8c5-1c5487abbb53 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:26.264545 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf1c4cf-7a96-4f8c-a025-c9fb9792bbc8 None admin] RBAC: Authorizing `identity:create_grant(project_id=7deb2c3dae214df7895b40e7339bd6ea, user_id=de64d498da78448394b0d4d1851a2789, role_id=bfe57c41a4da4b25b9b3e4761f220a46)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:26.266989 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf1c4cf-7a96-4f8c-a025-c9fb9792bbc8 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:26.288344 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 229/445] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:44:26 2022] PUT /identity/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/de64d498da78448394b0d4d1851a2789/roles/bfe57c41a4da4b25b9b3e4761f220a46 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:44:26.291033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9792c25c-a25b-4806-bacb-242003da918e None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:26.291585 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9792c25c-a25b-4806-bacb-242003da918e None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:26.292073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9792c25c-a25b-4806-bacb-242003da918e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:26.322643 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-9792c25c-a25b-4806-bacb-242003da918e None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:26.324730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-9792c25c-a25b-4806-bacb-242003da918e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:26.347854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:26.387247 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:26.409040 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4933ac5a-4968-4390-b8c5-1c5487abbb53 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:26.457294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 230/446] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:26.459214 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9792c25c-a25b-4806-bacb-242003da918e None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:26.485970 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9128bd-ad09-450d-be32-f21860089dc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:26.487379 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9128bd-ad09-450d-be32-f21860089dc6 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:26.488019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9128bd-ad09-450d-be32-f21860089dc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:26.488712 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9128bd-ad09-450d-be32-f21860089dc6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:26.509530 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9128bd-ad09-450d-be32-f21860089dc6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:26.512457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9128bd-ad09-450d-be32-f21860089dc6 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:26.540340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-4a9128bd-ad09-450d-be32-f21860089dc6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:26.540340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:26.540340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:26.540340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:26.540340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:26.540340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:26.540340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:26.540340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:26.540340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:26.540340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:26.540340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:26.540340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:26.540340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:26.540340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:26.540340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:26.540340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:26.542378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:26.542378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:26.542378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:26.542378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:26.542378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:26.542378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:26.542378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:26.542378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:26.542378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:26.542378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:26.542378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:26.542378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:26.542378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:26.542378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:26.542378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:26.542378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:26.542378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:26.544373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:26.544373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:26.544373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:26.546786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 217/447] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:26.548414 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9128bd-ad09-450d-be32-f21860089dc6 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:26.663516 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9792c25c-a25b-4806-bacb-242003da918e None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:26.734637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc815cba-52c8-42be-8138-0fdb40f0d262 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:26.735686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc815cba-52c8-42be-8138-0fdb40f0d262 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:26.736088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc815cba-52c8-42be-8138-0fdb40f0d262 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:26.736545 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc815cba-52c8-42be-8138-0fdb40f0d262 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:26.739712 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9128bd-ad09-450d-be32-f21860089dc6 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:26.761754 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc815cba-52c8-42be-8138-0fdb40f0d262 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:26.772386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-cc815cba-52c8-42be-8138-0fdb40f0d262 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:26.772386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:26.772386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:26.772386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:26.772386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:26.772386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:26.772386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:26.772386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:26.772386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:26.772386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:26.772386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:26.772386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:26.772386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:26.772386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:26.772386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:26.772386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:26.774542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:26.774542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:26.774542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:26.774542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:26.774542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:26.774542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:26.774542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:26.774542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:26.774542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:26.774542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:26.774542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:26.774542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:26.774542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:26.774542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:26.774542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:26.774542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:26.776382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:26.776382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:26.776382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:26.776382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:26.776382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:26.776382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:26.776382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:26.776382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:26.776382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:26.776382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:26.776382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:26.776382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:26.781812 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 231/448] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 16 00:44:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:26.783731 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc815cba-52c8-42be-8138-0fdb40f0d262 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:26.806633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:26.819083 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d35e0e-b013-4181-8ec5-1d088252e0a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:26.820376 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d35e0e-b013-4181-8ec5-1d088252e0a2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:26.821024 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d35e0e-b013-4181-8ec5-1d088252e0a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:26.821969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d35e0e-b013-4181-8ec5-1d088252e0a2 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:26.823339 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d35e0e-b013-4181-8ec5-1d088252e0a2 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:26.823737 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d35e0e-b013-4181-8ec5-1d088252e0a2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:26.825821 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d35e0e-b013-4181-8ec5-1d088252e0a2 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:26.860026 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:26.870824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 218/449] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Oct 16 00:44:26 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 323 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:26.872829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d35e0e-b013-4181-8ec5-1d088252e0a2 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:26.927567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc815cba-52c8-42be-8138-0fdb40f0d262 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:26.987989 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c68bae1-b5ac-4a2f-b6da-701c7b40f5bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:26.990132 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c68bae1-b5ac-4a2f-b6da-701c7b40f5bf None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:26.990978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c68bae1-b5ac-4a2f-b6da-701c7b40f5bf None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:26.991872 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c68bae1-b5ac-4a2f-b6da-701c7b40f5bf None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:26.993385 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c68bae1-b5ac-4a2f-b6da-701c7b40f5bf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:26.994186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c68bae1-b5ac-4a2f-b6da-701c7b40f5bf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:26.997043 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c68bae1-b5ac-4a2f-b6da-701c7b40f5bf None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:27.010597 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d35e0e-b013-4181-8ec5-1d088252e0a2 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:27.022010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 232/450] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:27.023576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c68bae1-b5ac-4a2f-b6da-701c7b40f5bf None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:27.058893 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694f8ee1-4b6f-48ee-9d37-87db55672f43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:27.059784 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694f8ee1-4b6f-48ee-9d37-87db55672f43 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:27.060064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694f8ee1-4b6f-48ee-9d37-87db55672f43 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:27.060351 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694f8ee1-4b6f-48ee-9d37-87db55672f43 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:27.061008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694f8ee1-4b6f-48ee-9d37-87db55672f43 None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:27.061313 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694f8ee1-4b6f-48ee-9d37-87db55672f43 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:27.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694f8ee1-4b6f-48ee-9d37-87db55672f43 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:27.078958 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 219/451] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 16 00:44:26 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:27.078958 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694f8ee1-4b6f-48ee-9d37-87db55672f43 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:27.143587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:27.182497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:27.191073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c68bae1-b5ac-4a2f-b6da-701c7b40f5bf None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:27.236505 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694f8ee1-4b6f-48ee-9d37-87db55672f43 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:27.256289 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f444b2-4ecb-4a76-8ff9-8c6aa2dc8433 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:27.257231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f444b2-4ecb-4a76-8ff9-8c6aa2dc8433 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:27.257745 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f444b2-4ecb-4a76-8ff9-8c6aa2dc8433 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:27.258245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f444b2-4ecb-4a76-8ff9-8c6aa2dc8433 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:27.281218 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60a9f00-323e-4337-a378-a9cae431a0c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:27.281880 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a9f00-323e-4337-a378-a9cae431a0c2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:27.282162 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a9f00-323e-4337-a378-a9cae431a0c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:27.282421 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a9f00-323e-4337-a378-a9cae431a0c2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:27.295153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60a9f00-323e-4337-a378-a9cae431a0c2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:27.300585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-b60a9f00-323e-4337-a378-a9cae431a0c2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:27.300585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:27.300585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:27.300585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:27.300585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:27.300585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:27.300585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:27.300585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:27.300585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:27.300585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:27.300585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:27.300585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:27.300585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:27.300585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:27.300585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:27.300585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:27.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:27.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:27.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:27.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:27.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:27.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:27.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:27.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:27.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:27.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:27.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:27.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:27.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:27.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:27.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:27.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:27.304019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:27.304019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:27.304019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:27.304019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:27.304019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:27.304019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:27.304019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:27.304019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:27.304019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:27.304019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:27.304019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:27.304019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:27.304019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 220/452] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 16 00:44:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:27.304019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60a9f00-323e-4337-a378-a9cae431a0c2 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:27.396578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f444b2-4ecb-4a76-8ff9-8c6aa2dc8433 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:27.399113 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f444b2-4ecb-4a76-8ff9-8c6aa2dc8433 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:27.466837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60a9f00-323e-4337-a378-a9cae431a0c2 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:27.506051 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:27.518185 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:27.543747 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5147efc8-07d7-4063-865d-95c0a4d2b854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:27.544883 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5147efc8-07d7-4063-865d-95c0a4d2b854 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:27.545386 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5147efc8-07d7-4063-865d-95c0a4d2b854 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:27.545870 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5147efc8-07d7-4063-865d-95c0a4d2b854 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:27.556307 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 233/453] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:44:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 533 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:44:27.561826 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f444b2-4ecb-4a76-8ff9-8c6aa2dc8433 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:27.563531 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5147efc8-07d7-4063-865d-95c0a4d2b854 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:27.565028 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5147efc8-07d7-4063-865d-95c0a4d2b854 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:27.587772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-5147efc8-07d7-4063-865d-95c0a4d2b854 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 16 00:44:27.587772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:27.587772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:27.587772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:27.587772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:27.587772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:27.587772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:27.587772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:27.587772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:27.587772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:27.587772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:27.587772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:27.587772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:44:27.587772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:44:27.587772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:44:27.587772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:27.588753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:27.589646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:27.589646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:27.589646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:44:27.589646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:44:27.589646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:44:27.589646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:44:27.589646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:44:27.589646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:44:27.589646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 16 00:44:27.589646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:27.590149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 221/454] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:44:27 2022] GET /identity/v3/users/glance => generated 84 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:27.591442 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5147efc8-07d7-4063-865d-95c0a4d2b854 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:27.704662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f444b2-4ecb-4a76-8ff9-8c6aa2dc8433 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:27.745314 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b317c4-b818-4b3d-b7b1-19f302a3b892 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:27.745971 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b317c4-b818-4b3d-b7b1-19f302a3b892 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:27.746246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b317c4-b818-4b3d-b7b1-19f302a3b892 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:27.746511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b317c4-b818-4b3d-b7b1-19f302a3b892 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:27.757549 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b317c4-b818-4b3d-b7b1-19f302a3b892 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:27.759256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b317c4-b818-4b3d-b7b1-19f302a3b892 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:27.768070 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5147efc8-07d7-4063-865d-95c0a4d2b854 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:27.775905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-c8b317c4-b818-4b3d-b7b1-19f302a3b892 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:27.775905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:27.775905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:27.775905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:27.775905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:27.775905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:27.775905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:27.775905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:27.775905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:27.775905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:27.775905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:27.775905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:27.775905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:27.775905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:27.775905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:27.775905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:27.776983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:27.776983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:27.776983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:27.776983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:27.776983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:27.776983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:27.776983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:27.776983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:27.776983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:27.776983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:27.776983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:27.776983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:27.776983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:27.776983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:27.776983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:27.776983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:27.776983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:27.777852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:27.777852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:27.777852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:27.777852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 234/455] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:27.778083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b317c4-b818-4b3d-b7b1-19f302a3b892 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:27.814415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:27.832096 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a53ed86-4a2f-42fa-bb9b-06ccf09dc40e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:27.833126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a53ed86-4a2f-42fa-bb9b-06ccf09dc40e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:27.833595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a53ed86-4a2f-42fa-bb9b-06ccf09dc40e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:27.834027 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a53ed86-4a2f-42fa-bb9b-06ccf09dc40e None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:27.835163 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a53ed86-4a2f-42fa-bb9b-06ccf09dc40e None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:27.835672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a53ed86-4a2f-42fa-bb9b-06ccf09dc40e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:27.838909 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a53ed86-4a2f-42fa-bb9b-06ccf09dc40e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:27.850666 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 222/456] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 16 00:44:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:27.851800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a53ed86-4a2f-42fa-bb9b-06ccf09dc40e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:27.931005 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:27.931258 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b317c4-b818-4b3d-b7b1-19f302a3b892 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:27.977295 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e87c92-5bb6-46f4-81b8-8a18ef989191 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:27.978469 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e87c92-5bb6-46f4-81b8-8a18ef989191 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:27.978749 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e87c92-5bb6-46f4-81b8-8a18ef989191 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:27.979071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e87c92-5bb6-46f4-81b8-8a18ef989191 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:27.999948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e87c92-5bb6-46f4-81b8-8a18ef989191 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:28.002041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e87c92-5bb6-46f4-81b8-8a18ef989191 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:28.014710 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a53ed86-4a2f-42fa-bb9b-06ccf09dc40e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:28.020615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-a1e87c92-5bb6-46f4-81b8-8a18ef989191 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:28.020615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:28.020615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:28.020615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:28.020615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:28.020615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:28.020615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:28.020615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:28.020615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:28.020615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:28.020615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:28.020615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:28.020615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:28.020615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:28.020615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:28.020615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:28.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:28.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:28.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:28.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:28.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:28.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:28.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:28.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:28.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:28.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:28.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:28.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:28.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:28.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:28.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:28.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:28.022356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:28.024063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:28.024063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:28.024063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:28.028938 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 235/457] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:28.029924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e87c92-5bb6-46f4-81b8-8a18ef989191 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:28.081740 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9e0e6a-fbf1-4a24-a692-5b9459edcab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:28.082801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9e0e6a-fbf1-4a24-a692-5b9459edcab5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:28.083355 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9e0e6a-fbf1-4a24-a692-5b9459edcab5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:28.083858 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9e0e6a-fbf1-4a24-a692-5b9459edcab5 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:28.085309 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9e0e6a-fbf1-4a24-a692-5b9459edcab5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:28.085875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9e0e6a-fbf1-4a24-a692-5b9459edcab5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:28.089263 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9e0e6a-fbf1-4a24-a692-5b9459edcab5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:28.119409 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:28.130063 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 223/458] 10.222.0.25 () {56 vars in 1137 bytes} [Sun Oct 16 00:44:27 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:28.131493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9e0e6a-fbf1-4a24-a692-5b9459edcab5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:28.161554 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e87c92-5bb6-46f4-81b8-8a18ef989191 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:28.205552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21432af3-d7a2-48c8-a9e4-668dec357a47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:28.206276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21432af3-d7a2-48c8-a9e4-668dec357a47 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:28.206554 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21432af3-d7a2-48c8-a9e4-668dec357a47 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:28.206832 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21432af3-d7a2-48c8-a9e4-668dec357a47 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:28.207530 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21432af3-d7a2-48c8-a9e4-668dec357a47 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:28.207837 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21432af3-d7a2-48c8-a9e4-668dec357a47 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:28.209403 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21432af3-d7a2-48c8-a9e4-668dec357a47 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:28.232336 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 236/459] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:28.232336 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21432af3-d7a2-48c8-a9e4-668dec357a47 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:28.338650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9e0e6a-fbf1-4a24-a692-5b9459edcab5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:28.341767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:28.397154 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21432af3-d7a2-48c8-a9e4-668dec357a47 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:28.410037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf1efb7-f822-406a-bad9-ceb688f2fb81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:28.411186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf1efb7-f822-406a-bad9-ceb688f2fb81 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:28.411694 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf1efb7-f822-406a-bad9-ceb688f2fb81 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:28.412189 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf1efb7-f822-406a-bad9-ceb688f2fb81 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:28.433704 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf1efb7-f822-406a-bad9-ceb688f2fb81 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:28.436499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf1efb7-f822-406a-bad9-ceb688f2fb81 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:28.460675 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:28.467084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-2cf1efb7-f822-406a-bad9-ceb688f2fb81 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:28.467084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:28.467084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:28.467084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:28.467084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:28.467084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:28.467084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:28.467084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:28.467084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:28.467084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:28.467084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:28.467084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:28.467084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:28.467084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:28.467084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:28.467084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:28.468064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:28.468064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:28.468064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:28.468064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:28.468064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:28.468064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:28.468064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:28.468064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:28.468064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:28.468064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:28.468064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:28.468064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:28.468064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:28.468064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:28.468064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:28.468064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:28.468064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:28.468921 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:28.468921 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:28.468921 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:28.474331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 224/460] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 339 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:28.474331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf1efb7-f822-406a-bad9-ceb688f2fb81 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:28.474602 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662cbe29-004f-4e64-89d2-ad675dcd9784 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:28.474602 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662cbe29-004f-4e64-89d2-ad675dcd9784 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:28.474602 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662cbe29-004f-4e64-89d2-ad675dcd9784 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:28.474602 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662cbe29-004f-4e64-89d2-ad675dcd9784 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:28.475729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662cbe29-004f-4e64-89d2-ad675dcd9784 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:28.476893 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662cbe29-004f-4e64-89d2-ad675dcd9784 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:28.481279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-662cbe29-004f-4e64-89d2-ad675dcd9784 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:28.509760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 237/461] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:28.509760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662cbe29-004f-4e64-89d2-ad675dcd9784 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:28.675183 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf1efb7-f822-406a-bad9-ceb688f2fb81 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:28.703382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662cbe29-004f-4e64-89d2-ad675dcd9784 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:28.743826 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b512a733-84e6-4a0a-b68e-45d77db08f8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:28.745223 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b512a733-84e6-4a0a-b68e-45d77db08f8e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:28.745896 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b512a733-84e6-4a0a-b68e-45d77db08f8e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:28.747033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b512a733-84e6-4a0a-b68e-45d77db08f8e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:28.768554 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470b8f78-73c5-45ea-9621-b8d3b849871f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:28.769579 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470b8f78-73c5-45ea-9621-b8d3b849871f None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:28.770086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470b8f78-73c5-45ea-9621-b8d3b849871f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:28.770599 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470b8f78-73c5-45ea-9621-b8d3b849871f None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:28.787216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470b8f78-73c5-45ea-9621-b8d3b849871f None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:28.790462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470b8f78-73c5-45ea-9621-b8d3b849871f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:28.805442 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:28.817796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-470b8f78-73c5-45ea-9621-b8d3b849871f None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 16 00:44:28.817796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:28.817796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:28.817796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:28.817796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:28.817796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:28.817796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:28.817796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:28.817796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:28.817796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:28.817796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:28.817796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:28.817796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:44:28.817796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:44:28.817796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:44:28.817796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:28.818787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:28.819716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:28.819716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:28.819716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:44:28.819716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:44:28.819716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:44:28.819716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:44:28.819716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:44:28.819716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:44:28.819716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 16 00:44:28.819716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:28.828170 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 238/462] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 16 00:44:28 2022] GET /identity/v3/users/neutron => generated 85 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:28.830284 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470b8f78-73c5-45ea-9621-b8d3b849871f None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:28.862792 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b512a733-84e6-4a0a-b68e-45d77db08f8e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:28.864320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b512a733-84e6-4a0a-b68e-45d77db08f8e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:28.938611 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:28.984963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 225/463] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:44:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 515 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:44:28.985866 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b512a733-84e6-4a0a-b68e-45d77db08f8e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:29.004529 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470b8f78-73c5-45ea-9621-b8d3b849871f None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:29.056396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:29.058248 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86792c2b-564e-49dc-952d-9a9a467ff975 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:29.059095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86792c2b-564e-49dc-952d-9a9a467ff975 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:29.059476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86792c2b-564e-49dc-952d-9a9a467ff975 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:29.059787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86792c2b-564e-49dc-952d-9a9a467ff975 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:29.060453 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86792c2b-564e-49dc-952d-9a9a467ff975 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:29.060812 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86792c2b-564e-49dc-952d-9a9a467ff975 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:29.062375 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86792c2b-564e-49dc-952d-9a9a467ff975 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:29.075864 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 239/464] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:29.076755 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86792c2b-564e-49dc-952d-9a9a467ff975 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:29.108826 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b512a733-84e6-4a0a-b68e-45d77db08f8e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:29.146163 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9487623-3147-408f-9b93-4ac175a82de2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:29.146815 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9487623-3147-408f-9b93-4ac175a82de2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:29.147128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9487623-3147-408f-9b93-4ac175a82de2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:29.147404 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9487623-3147-408f-9b93-4ac175a82de2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:29.155287 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9487623-3147-408f-9b93-4ac175a82de2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:29.156759 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9487623-3147-408f-9b93-4ac175a82de2 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:29.169823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-c9487623-3147-408f-9b93-4ac175a82de2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:29.169823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:29.169823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:29.169823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:29.169823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:29.169823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:29.169823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:29.169823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:29.169823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:29.169823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:29.169823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:29.169823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:29.169823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:44:29.169823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:44:29.169823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:44:29.171422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:44:29.171422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:29.171422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:29.171422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:29.171422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:29.171422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:29.171422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:29.171422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:29.171422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:29.171422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:29.171422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:29.171422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:29.171422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:29.171422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:29.171422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:29.171422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:29.171422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:29.172438 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:44:29.172438 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:44:29.172438 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:44:29.172438 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:44:29.172438 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:44:29.172438 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:44:29.172438 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:29.172438 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:29.175274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 226/465] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:29.176910 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2ddf47-cd3c-44c8-aacf-3a166a5d64c8 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:29.177297 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2ddf47-cd3c-44c8-aacf-3a166a5d64c8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:29.177663 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2ddf47-cd3c-44c8-aacf-3a166a5d64c8 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:29.179251 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 227/466] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:29.180192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2ddf47-cd3c-44c8-aacf-3a166a5d64c8 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:29.227390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86792c2b-564e-49dc-952d-9a9a467ff975 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:29.249821 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:29.279121 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc040e48-310b-4327-816e-f7277fe300dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:29.279744 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc040e48-310b-4327-816e-f7277fe300dc None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:29.280028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc040e48-310b-4327-816e-f7277fe300dc None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:29.280305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc040e48-310b-4327-816e-f7277fe300dc None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:29.281108 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc040e48-310b-4327-816e-f7277fe300dc None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:29.281417 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc040e48-310b-4327-816e-f7277fe300dc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:29.282967 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc040e48-310b-4327-816e-f7277fe300dc None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:29.303926 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 240/467] 10.222.0.25 () {56 vars in 1139 bytes} [Sun Oct 16 00:44:29 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:29.305136 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc040e48-310b-4327-816e-f7277fe300dc None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:29.317834 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2ddf47-cd3c-44c8-aacf-3a166a5d64c8 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:29.337574 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:29.374031 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499e52a9-f270-4f39-9139-bbff2fc85a1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:29.375135 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499e52a9-f270-4f39-9139-bbff2fc85a1d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:29.375636 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499e52a9-f270-4f39-9139-bbff2fc85a1d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:29.376112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499e52a9-f270-4f39-9139-bbff2fc85a1d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:29.393792 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499e52a9-f270-4f39-9139-bbff2fc85a1d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:29.395655 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499e52a9-f270-4f39-9139-bbff2fc85a1d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:29.417670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-499e52a9-f270-4f39-9139-bbff2fc85a1d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:29.417670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:29.417670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:29.417670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:29.417670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:29.417670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:29.417670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:29.417670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:29.417670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:29.417670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:29.417670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:29.417670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:29.417670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:29.417670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:29.417670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:29.417670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:29.419419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:29.419419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:29.419419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:29.419419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:29.419419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:29.419419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:29.419419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:29.419419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:29.419419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:29.419419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:29.419419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:29.419419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:29.419419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:29.419419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:29.419419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:29.419419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:29.419419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:29.420630 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:29.420630 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:29.420630 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:29.427261 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 228/468] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:29.427261 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499e52a9-f270-4f39-9139-bbff2fc85a1d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:29.452541 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc040e48-310b-4327-816e-f7277fe300dc None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:29.494180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4fe8e4-e172-48f2-a873-4f272089c9be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:29.494765 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4fe8e4-e172-48f2-a873-4f272089c9be None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:29.495078 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4fe8e4-e172-48f2-a873-4f272089c9be None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:29.495358 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4fe8e4-e172-48f2-a873-4f272089c9be None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:29.503618 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4fe8e4-e172-48f2-a873-4f272089c9be None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:29.505091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4fe8e4-e172-48f2-a873-4f272089c9be None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:29.518700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-bd4fe8e4-e172-48f2-a873-4f272089c9be None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 16 00:44:29.518700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:29.518700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:29.518700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:29.518700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:29.518700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:29.518700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:29.518700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:29.518700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:29.518700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:29.518700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:29.518700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:29.518700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:44:29.518700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:44:29.518700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:44:29.518700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:29.520320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:29.522109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:29.522109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:29.522109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:44:29.522109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:44:29.522109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:44:29.522109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:44:29.522109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:44:29.522109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:44:29.522109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 16 00:44:29.522109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:29.522109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 241/469] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:44:29 2022] GET /identity/v3/users/cinder => generated 84 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:29.522109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea899ef4-49a0-4d9e-8868-ce5fbea38a30 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:29.522109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea899ef4-49a0-4d9e-8868-ce5fbea38a30 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:29.523481 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea899ef4-49a0-4d9e-8868-ce5fbea38a30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:29.541888 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-ea899ef4-49a0-4d9e-8868-ce5fbea38a30 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:29.543253 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-ea899ef4-49a0-4d9e-8868-ce5fbea38a30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:29.558424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499e52a9-f270-4f39-9139-bbff2fc85a1d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:29.578132 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:29.614393 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859f2a6c-7cab-40d5-aa29-0d2d96b80ad5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:29.615478 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859f2a6c-7cab-40d5-aa29-0d2d96b80ad5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:29.615974 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859f2a6c-7cab-40d5-aa29-0d2d96b80ad5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:29.616460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859f2a6c-7cab-40d5-aa29-0d2d96b80ad5 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:29.617729 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859f2a6c-7cab-40d5-aa29-0d2d96b80ad5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:29.618830 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859f2a6c-7cab-40d5-aa29-0d2d96b80ad5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:29.621601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859f2a6c-7cab-40d5-aa29-0d2d96b80ad5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:29.638892 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 242/470] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:29.638892 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea899ef4-49a0-4d9e-8868-ce5fbea38a30 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:29.644924 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 229/471] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:44:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:29.647457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859f2a6c-7cab-40d5-aa29-0d2d96b80ad5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:29.682453 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:29.771671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859f2a6c-7cab-40d5-aa29-0d2d96b80ad5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:29.810854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea899ef4-49a0-4d9e-8868-ce5fbea38a30 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:29.812528 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a512db7c-20b3-46d9-a50c-fbfd3f5b8acd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:29.813947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a512db7c-20b3-46d9-a50c-fbfd3f5b8acd None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:29.815448 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a512db7c-20b3-46d9-a50c-fbfd3f5b8acd None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:29.815907 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a512db7c-20b3-46d9-a50c-fbfd3f5b8acd None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:29.817040 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a512db7c-20b3-46d9-a50c-fbfd3f5b8acd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:29.817546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a512db7c-20b3-46d9-a50c-fbfd3f5b8acd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:29.819960 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a512db7c-20b3-46d9-a50c-fbfd3f5b8acd None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:29.844345 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 230/472] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:29.844345 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a512db7c-20b3-46d9-a50c-fbfd3f5b8acd None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:29.873064 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60581b73-c956-4dbc-a719-10f473050e1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:29.873788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60581b73-c956-4dbc-a719-10f473050e1b None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:29.874196 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60581b73-c956-4dbc-a719-10f473050e1b None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:29.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60581b73-c956-4dbc-a719-10f473050e1b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:29.912321 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:29.980003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:29.988808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60581b73-c956-4dbc-a719-10f473050e1b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:29.990939 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60581b73-c956-4dbc-a719-10f473050e1b None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:30.002284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a512db7c-20b3-46d9-a50c-fbfd3f5b8acd None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:30.043738 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76426b3-27e5-464c-8515-ae50dc60efe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:30.044424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76426b3-27e5-464c-8515-ae50dc60efe3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:30.044709 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76426b3-27e5-464c-8515-ae50dc60efe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:30.044981 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76426b3-27e5-464c-8515-ae50dc60efe3 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:30.045716 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76426b3-27e5-464c-8515-ae50dc60efe3 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:30.046021 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76426b3-27e5-464c-8515-ae50dc60efe3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:30.047523 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76426b3-27e5-464c-8515-ae50dc60efe3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:30.066977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 231/473] 10.222.0.25 () {56 vars in 1137 bytes} [Sun Oct 16 00:44:29 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:30.067844 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76426b3-27e5-464c-8515-ae50dc60efe3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:30.115887 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 243/474] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:44:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 482 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:44:30.117974 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60581b73-c956-4dbc-a719-10f473050e1b None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:30.190036 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76426b3-27e5-464c-8515-ae50dc60efe3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:30.213505 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:30.240516 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7699efa0-9627-429f-90c7-c6104bef63ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:30.241548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7699efa0-9627-429f-90c7-c6104bef63ec None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:30.242029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7699efa0-9627-429f-90c7-c6104bef63ec None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:30.242518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7699efa0-9627-429f-90c7-c6104bef63ec None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:30.244053 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7699efa0-9627-429f-90c7-c6104bef63ec None admin] RBAC: Adding query filter params (role.id=bfe57c41a4da4b25b9b3e4761f220a46, scope.project.id=7deb2c3dae214df7895b40e7339bd6ea, user.id=4eba26813e154595a12a67fe4be5eeff) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:30.244537 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7699efa0-9627-429f-90c7-c6104bef63ec None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:30.247123 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7699efa0-9627-429f-90c7-c6104bef63ec None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:30.270902 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 232/475] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:44:30 2022] GET /identity/v3/role_assignments?user.id=4eba26813e154595a12a67fe4be5eeff&scope.project.id=7deb2c3dae214df7895b40e7339bd6ea&role.id=bfe57c41a4da4b25b9b3e4761f220a46 => generated 608 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:30.271159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60581b73-c956-4dbc-a719-10f473050e1b None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:30.271309 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7699efa0-9627-429f-90c7-c6104bef63ec None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:30.280410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:30.308949 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487675af-1ee4-4a0b-9cf6-bb085dec6fe8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:30.309604 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487675af-1ee4-4a0b-9cf6-bb085dec6fe8 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:30.309964 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487675af-1ee4-4a0b-9cf6-bb085dec6fe8 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:30.310247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487675af-1ee4-4a0b-9cf6-bb085dec6fe8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:30.318920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487675af-1ee4-4a0b-9cf6-bb085dec6fe8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:30.320572 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487675af-1ee4-4a0b-9cf6-bb085dec6fe8 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:30.338769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-487675af-1ee4-4a0b-9cf6-bb085dec6fe8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:30.338769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:30.338769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:30.338769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:30.338769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:30.338769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:30.338769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:30.338769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:30.338769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:30.338769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:30.338769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:30.338769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:30.338769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:44:30.338769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:44:30.338769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:44:30.340451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:44:30.340451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:30.340451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:30.340451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:30.340451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:30.340451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:30.340451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:30.340451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:30.340451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:30.340451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:30.340451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:30.340451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:30.340451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:30.340451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:30.340451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:30.340451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:30.340451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:30.343245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:44:30.343245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:44:30.343245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:44:30.343245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:44:30.343245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:44:30.343245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:44:30.343245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:30.343245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:30.343245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 244/476] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:30.343245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487675af-1ee4-4a0b-9cf6-bb085dec6fe8 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:30.425453 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7699efa0-9627-429f-90c7-c6104bef63ec None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:30.489308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f80fe42-6b45-437f-a58b-cafeb393e4be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:30.489966 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f80fe42-6b45-437f-a58b-cafeb393e4be None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:30.490464 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f80fe42-6b45-437f-a58b-cafeb393e4be None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:30.490737 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f80fe42-6b45-437f-a58b-cafeb393e4be None admin] PATH_INFO: `/v3/roles/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:30.504462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f80fe42-6b45-437f-a58b-cafeb393e4be None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:30.509941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-8f80fe42-6b45-437f-a58b-cafeb393e4be None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:30.509941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:30.509941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:30.509941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:30.509941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:30.509941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:30.509941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:30.509941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:30.509941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:30.509941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:30.509941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:30.509941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:30.509941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:30.509941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:30.509941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:30.509941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:30.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:30.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:30.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:30.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:30.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:30.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:30.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:30.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:30.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:30.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:30.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:30.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:30.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:30.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:30.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:30.511145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:30.512192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:30.512192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:30.512192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:30.512192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:30.512192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:30.512192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:30.512192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:30.512192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:30.512192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:30.512192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:30.512192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:30.512192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:30.512192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 233/477] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 16 00:44:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:30.512192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f80fe42-6b45-437f-a58b-cafeb393e4be None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:30.532047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487675af-1ee4-4a0b-9cf6-bb085dec6fe8 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:30.561344 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:30.584610 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4a78a4-fe44-4ac1-be86-430b1c3288db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:30.585232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4a78a4-fe44-4ac1-be86-430b1c3288db None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:30.585501 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4a78a4-fe44-4ac1-be86-430b1c3288db None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:30.585815 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4a78a4-fe44-4ac1-be86-430b1c3288db None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:30.617295 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:30.677169 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4a78a4-fe44-4ac1-be86-430b1c3288db None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:30.678661 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4a78a4-fe44-4ac1-be86-430b1c3288db None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:30.684031 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f80fe42-6b45-437f-a58b-cafeb393e4be None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:30.734761 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5f8bc9-558b-4e33-8b5f-fba720ac4a5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:30.735431 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5f8bc9-558b-4e33-8b5f-fba720ac4a5f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:30.735716 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5f8bc9-558b-4e33-8b5f-fba720ac4a5f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:30.735974 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5f8bc9-558b-4e33-8b5f-fba720ac4a5f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:30.748863 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5f8bc9-558b-4e33-8b5f-fba720ac4a5f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:30.750347 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5f8bc9-558b-4e33-8b5f-fba720ac4a5f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:30.768205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-ce5f8bc9-558b-4e33-8b5f-fba720ac4a5f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:30.768205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:30.768205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:30.768205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:30.768205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:30.768205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:30.768205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:30.768205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:30.768205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:30.768205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:30.768205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:30.768205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:30.768205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:30.768205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:30.768205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:30.768205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:30.770170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:30.770170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:30.770170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:30.770170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:30.770170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:30.770170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:30.770170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:30.770170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:30.770170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:30.770170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:30.770170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:30.770170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:30.770170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:30.770170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:30.770170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:30.770170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:30.770170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:30.771856 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:30.771856 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:30.771856 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:30.771856 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 234/478] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:30.771856 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5f8bc9-558b-4e33-8b5f-fba720ac4a5f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:30.865157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 245/479] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:44:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 524 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:44:30.867079 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4a78a4-fe44-4ac1-be86-430b1c3288db None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:30.905505 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:30.938837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5f8bc9-558b-4e33-8b5f-fba720ac4a5f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:30.966898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:30.986781 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afd8811-55a3-4603-887d-7da64b3976f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:30.987569 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afd8811-55a3-4603-887d-7da64b3976f8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:30.987939 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afd8811-55a3-4603-887d-7da64b3976f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:30.988390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afd8811-55a3-4603-887d-7da64b3976f8 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:30.989288 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afd8811-55a3-4603-887d-7da64b3976f8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:30.989691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afd8811-55a3-4603-887d-7da64b3976f8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:30.992187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afd8811-55a3-4603-887d-7da64b3976f8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:31.008330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 235/480] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:44:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:31.010503 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afd8811-55a3-4603-887d-7da64b3976f8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:31.034192 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4a78a4-fe44-4ac1-be86-430b1c3288db None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:31.079197 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b924ccc0-bfa2-4882-9f88-2fdbe45b4002 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:31.079822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b924ccc0-bfa2-4882-9f88-2fdbe45b4002 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:31.080351 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b924ccc0-bfa2-4882-9f88-2fdbe45b4002 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:31.081100 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b924ccc0-bfa2-4882-9f88-2fdbe45b4002 None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:31.081792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b924ccc0-bfa2-4882-9f88-2fdbe45b4002 None admin] RBAC: Adding query filter params (name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:31.082092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b924ccc0-bfa2-4882-9f88-2fdbe45b4002 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:31.083864 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b924ccc0-bfa2-4882-9f88-2fdbe45b4002 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:31.094221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 246/481] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 16 00:44:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:31.095721 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b924ccc0-bfa2-4882-9f88-2fdbe45b4002 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:31.159579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afd8811-55a3-4603-887d-7da64b3976f8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:31.196666 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:31.205425 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b285ff69-e26c-49a9-9b70-c8c55d7d5f46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:31.206059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b285ff69-e26c-49a9-9b70-c8c55d7d5f46 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:31.206347 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b285ff69-e26c-49a9-9b70-c8c55d7d5f46 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:31.206628 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b285ff69-e26c-49a9-9b70-c8c55d7d5f46 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:31.207360 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b285ff69-e26c-49a9-9b70-c8c55d7d5f46 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:31.207671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b285ff69-e26c-49a9-9b70-c8c55d7d5f46 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:31.209072 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b285ff69-e26c-49a9-9b70-c8c55d7d5f46 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:31.222855 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 236/482] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:31.223818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b285ff69-e26c-49a9-9b70-c8c55d7d5f46 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:31.236700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b924ccc0-bfa2-4882-9f88-2fdbe45b4002 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:31.274549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:31.306067 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af21fad7-c387-455d-b7d3-b91d10310cfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:31.307192 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af21fad7-c387-455d-b7d3-b91d10310cfb None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:31.307705 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af21fad7-c387-455d-b7d3-b91d10310cfb None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:31.308217 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af21fad7-c387-455d-b7d3-b91d10310cfb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:31.324546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af21fad7-c387-455d-b7d3-b91d10310cfb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:31.326041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af21fad7-c387-455d-b7d3-b91d10310cfb None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:31.348408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-af21fad7-c387-455d-b7d3-b91d10310cfb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:31.348408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:31.348408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:31.348408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:31.348408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:31.348408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:31.348408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:31.348408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:31.348408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:31.348408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:31.348408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:31.348408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:31.348408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:31.348408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:31.348408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:31.348408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:31.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:31.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:31.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:31.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:31.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:31.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:31.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:31.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:31.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:31.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:31.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:31.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:31.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:31.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:31.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:31.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:31.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:31.350280 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:31.350280 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:31.350280 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:31.350443 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 247/483] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:31.351906 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af21fad7-c387-455d-b7d3-b91d10310cfb None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:31.361990 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b285ff69-e26c-49a9-9b70-c8c55d7d5f46 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:31.404566 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362746c5-0981-422d-a5ca-280c2debf18e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:31.406668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362746c5-0981-422d-a5ca-280c2debf18e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:31.407023 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362746c5-0981-422d-a5ca-280c2debf18e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:31.407331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362746c5-0981-422d-a5ca-280c2debf18e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:31.408603 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362746c5-0981-422d-a5ca-280c2debf18e None admin] RBAC: Adding query filter params (role.id=bfe57c41a4da4b25b9b3e4761f220a46, scope.project.id=7deb2c3dae214df7895b40e7339bd6ea, user.id=78473c29cef449b89bc9185de67f1bf7) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:31.408925 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362746c5-0981-422d-a5ca-280c2debf18e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:31.411041 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362746c5-0981-422d-a5ca-280c2debf18e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:31.421387 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 237/484] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:44:31 2022] GET /identity/v3/role_assignments?user.id=78473c29cef449b89bc9185de67f1bf7&scope.project.id=7deb2c3dae214df7895b40e7339bd6ea&role.id=bfe57c41a4da4b25b9b3e4761f220a46 => generated 608 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:31.422233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362746c5-0981-422d-a5ca-280c2debf18e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:31.517255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af21fad7-c387-455d-b7d3-b91d10310cfb None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:31.566873 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:31.575189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b158b2-9e41-4451-9e99-f44961b1787d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:31.575970 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b158b2-9e41-4451-9e99-f44961b1787d None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:31.576261 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b158b2-9e41-4451-9e99-f44961b1787d None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:31.576537 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b158b2-9e41-4451-9e99-f44961b1787d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:31.582242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362746c5-0981-422d-a5ca-280c2debf18e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:31.586168 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:31.591421 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b158b2-9e41-4451-9e99-f44961b1787d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:31.592956 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b158b2-9e41-4451-9e99-f44961b1787d None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:31.617993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-54b158b2-9e41-4451-9e99-f44961b1787d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:31.617993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:31.617993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:31.617993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:31.617993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:31.617993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:31.617993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:31.617993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:31.617993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:31.617993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:31.617993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:31.617993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:31.617993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:31.617993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:31.617993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:31.617993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:31.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:31.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:31.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:31.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:31.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:31.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:31.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:31.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:31.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:31.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:31.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:31.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:31.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:31.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:31.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:31.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:31.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:31.620734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:31.620734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:31.620734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:31.620734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 248/485] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:31.622183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b158b2-9e41-4451-9e99-f44961b1787d None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:31.645921 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff94d74e-ac9b-455e-9d52-4437e72ce76d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:31.647458 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff94d74e-ac9b-455e-9d52-4437e72ce76d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:31.648502 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff94d74e-ac9b-455e-9d52-4437e72ce76d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:31.649001 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff94d74e-ac9b-455e-9d52-4437e72ce76d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:31.664543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff94d74e-ac9b-455e-9d52-4437e72ce76d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:31.667469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff94d74e-ac9b-455e-9d52-4437e72ce76d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:31.681551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-ff94d74e-ac9b-455e-9d52-4437e72ce76d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:31.681551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:31.681551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:31.681551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:31.681551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:31.681551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:31.681551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:31.681551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:31.681551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:31.681551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:31.681551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:31.681551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:31.681551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:44:31.681551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:44:31.681551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:44:31.683410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:44:31.683410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:31.683410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:31.683410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:31.683410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:31.683410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:31.683410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:31.683410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:31.683410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:31.683410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:31.683410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:31.683410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:31.683410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:31.683410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:31.683410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:31.683410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:31.683410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:31.685109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:44:31.685109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:44:31.685109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:44:31.685109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:44:31.685109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:44:31.685109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:44:31.685109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:31.685109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:31.687113 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 238/486] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:31.688141 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff94d74e-ac9b-455e-9d52-4437e72ce76d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:31.790917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b158b2-9e41-4451-9e99-f44961b1787d None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:31.832531 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec30322-b086-4aa6-9a70-c458fda5e2a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:31.833181 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec30322-b086-4aa6-9a70-c458fda5e2a3 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:31.833463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec30322-b086-4aa6-9a70-c458fda5e2a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:31.833731 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec30322-b086-4aa6-9a70-c458fda5e2a3 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:31.834373 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec30322-b086-4aa6-9a70-c458fda5e2a3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:31.834675 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec30322-b086-4aa6-9a70-c458fda5e2a3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:31.836100 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec30322-b086-4aa6-9a70-c458fda5e2a3 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:31.861273 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 249/487] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:31.862228 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec30322-b086-4aa6-9a70-c458fda5e2a3 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:31.903099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff94d74e-ac9b-455e-9d52-4437e72ce76d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:31.922728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:31.957114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ae72f9-6cf3-4ef7-a45f-d9ab0f7c206e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:31.957770 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ae72f9-6cf3-4ef7-a45f-d9ab0f7c206e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:31.958017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ae72f9-6cf3-4ef7-a45f-d9ab0f7c206e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:31.958296 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ae72f9-6cf3-4ef7-a45f-d9ab0f7c206e None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:31.959728 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ae72f9-6cf3-4ef7-a45f-d9ab0f7c206e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:31.960070 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ae72f9-6cf3-4ef7-a45f-d9ab0f7c206e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:31.961511 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ae72f9-6cf3-4ef7-a45f-d9ab0f7c206e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:31.982972 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 239/488] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:31.983968 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ae72f9-6cf3-4ef7-a45f-d9ab0f7c206e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:31.995851 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec30322-b086-4aa6-9a70-c458fda5e2a3 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:32.021622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:32.049840 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4c24b7-c7b7-40a9-b5bb-337038b7ae44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:32.050787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4c24b7-c7b7-40a9-b5bb-337038b7ae44 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:32.051112 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4c24b7-c7b7-40a9-b5bb-337038b7ae44 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:32.051394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4c24b7-c7b7-40a9-b5bb-337038b7ae44 None admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:32.052144 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4c24b7-c7b7-40a9-b5bb-337038b7ae44 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:32.052459 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4c24b7-c7b7-40a9-b5bb-337038b7ae44 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:32.054652 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4c24b7-c7b7-40a9-b5bb-337038b7ae44 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:32.078578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 250/489] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:44:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:32.078578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4c24b7-c7b7-40a9-b5bb-337038b7ae44 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:32.122351 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ae72f9-6cf3-4ef7-a45f-d9ab0f7c206e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:32.166794 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cedcb58-4741-44c1-ba40-f47edd7d2bca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:32.168222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cedcb58-4741-44c1-ba40-f47edd7d2bca None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:32.168825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cedcb58-4741-44c1-ba40-f47edd7d2bca None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:32.169379 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cedcb58-4741-44c1-ba40-f47edd7d2bca None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:32.179628 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cedcb58-4741-44c1-ba40-f47edd7d2bca None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:32.183974 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cedcb58-4741-44c1-ba40-f47edd7d2bca None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:32.208944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-8cedcb58-4741-44c1-ba40-f47edd7d2bca None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:32.208944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:32.208944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:32.208944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:32.208944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:32.208944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:32.208944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:32.208944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:32.208944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:32.208944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:32.208944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:32.208944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:32.208944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:44:32.208944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:44:32.208944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:44:32.210324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:44:32.210324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:32.210324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:32.210324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:32.210324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:32.210324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:32.210324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:32.210324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:32.210324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:32.210324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:32.210324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:32.210324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:32.210324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:32.210324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:32.210324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:32.210324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:32.210324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:32.213155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:44:32.213155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:44:32.213155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:44:32.213155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:44:32.213155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:44:32.213155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:44:32.213155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:32.213155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:32.217551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 240/490] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:32.219381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cedcb58-4741-44c1-ba40-f47edd7d2bca None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:32.259496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4c24b7-c7b7-40a9-b5bb-337038b7ae44 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:32.263584 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:32.316918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adde37b-e435-4acd-b0ea-059187c7cf94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:32.318621 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adde37b-e435-4acd-b0ea-059187c7cf94 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:32.319246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adde37b-e435-4acd-b0ea-059187c7cf94 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:32.319811 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adde37b-e435-4acd-b0ea-059187c7cf94 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:32.338351 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9adde37b-e435-4acd-b0ea-059187c7cf94 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:32.341355 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9adde37b-e435-4acd-b0ea-059187c7cf94 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:32.357586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-9adde37b-e435-4acd-b0ea-059187c7cf94 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 16 00:44:32.357586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:32.357586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:32.357586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:32.357586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:32.357586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:32.357586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:32.357586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:32.357586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:32.357586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:32.357586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:32.357586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:32.357586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:44:32.357586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:44:32.357586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:44:32.357586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:32.359627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:32.361423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:32.361423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:32.361423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:44:32.361423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:44:32.361423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:44:32.361423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:44:32.361423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:44:32.361423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:44:32.361423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 16 00:44:32.361423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:32.361423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 251/491] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:44:32 2022] GET /identity/v3/users/swift => generated 83 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:32.361423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adde37b-e435-4acd-b0ea-059187c7cf94 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:32.397922 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:32.429759 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cedcb58-4741-44c1-ba40-f47edd7d2bca None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:32.487880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adde37b-e435-4acd-b0ea-059187c7cf94 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:32.499022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c983a6-08cc-4e1e-b8b9-fc0a932f85dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:32.500697 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c983a6-08cc-4e1e-b8b9-fc0a932f85dc None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:32.501469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c983a6-08cc-4e1e-b8b9-fc0a932f85dc None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:32.502215 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c983a6-08cc-4e1e-b8b9-fc0a932f85dc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:32.504397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c983a6-08cc-4e1e-b8b9-fc0a932f85dc None admin] RBAC: Adding query filter params (role.id=bfe57c41a4da4b25b9b3e4761f220a46, scope.project.id=7deb2c3dae214df7895b40e7339bd6ea, user.id=f4eb1a640186411097cb1e3bb9e92020) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:32.505298 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c983a6-08cc-4e1e-b8b9-fc0a932f85dc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:32.509187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c983a6-08cc-4e1e-b8b9-fc0a932f85dc None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:32.527247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672e7c76-a9db-4598-b036-126ff9176414 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:32.527677 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672e7c76-a9db-4598-b036-126ff9176414 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:32.527946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672e7c76-a9db-4598-b036-126ff9176414 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:32.528223 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672e7c76-a9db-4598-b036-126ff9176414 None admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:32.528617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 241/492] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:44:32 2022] GET /identity/v3/role_assignments?user.id=f4eb1a640186411097cb1e3bb9e92020&scope.project.id=7deb2c3dae214df7895b40e7339bd6ea&role.id=bfe57c41a4da4b25b9b3e4761f220a46 => generated 608 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:32.528870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672e7c76-a9db-4598-b036-126ff9176414 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:32.529177 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672e7c76-a9db-4598-b036-126ff9176414 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:32.530664 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672e7c76-a9db-4598-b036-126ff9176414 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:32.531314 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c983a6-08cc-4e1e-b8b9-fc0a932f85dc None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:32.545557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 252/493] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:44:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:32.554385 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672e7c76-a9db-4598-b036-126ff9176414 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:32.615503 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:32.759962 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672e7c76-a9db-4598-b036-126ff9176414 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:32.784979 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c983a6-08cc-4e1e-b8b9-fc0a932f85dc None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:32.820943 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75894369-d05f-4ea8-bcfa-56e17478609a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:32.821646 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75894369-d05f-4ea8-bcfa-56e17478609a None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:32.822096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75894369-d05f-4ea8-bcfa-56e17478609a None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:32.822390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75894369-d05f-4ea8-bcfa-56e17478609a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:32.823129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75894369-d05f-4ea8-bcfa-56e17478609a None admin] RBAC: Adding query filter params (role.id=bfe57c41a4da4b25b9b3e4761f220a46, scope.project.id=7deb2c3dae214df7895b40e7339bd6ea, user.id=81fa2042309c48e1859b601d8bee1d5b) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:32.823623 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75894369-d05f-4ea8-bcfa-56e17478609a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:32.825134 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75894369-d05f-4ea8-bcfa-56e17478609a None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:32.837294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 253/494] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:44:32 2022] GET /identity/v3/role_assignments?user.id=81fa2042309c48e1859b601d8bee1d5b&scope.project.id=7deb2c3dae214df7895b40e7339bd6ea&role.id=bfe57c41a4da4b25b9b3e4761f220a46 => generated 608 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:32.837294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d26939f-5817-4f9c-90a9-2eeed1b748af None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:32.837294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d26939f-5817-4f9c-90a9-2eeed1b748af None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:32.837294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d26939f-5817-4f9c-90a9-2eeed1b748af None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:32.837294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 254/495] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:32.855709 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7bc817-3c1d-42fd-861b-08e4a2f630eb None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:32.856095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7bc817-3c1d-42fd-861b-08e4a2f630eb None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:32.856453 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7bc817-3c1d-42fd-861b-08e4a2f630eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:32.880222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec8619e-99dc-40f0-8efd-b7e1ca01614c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:32.880973 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec8619e-99dc-40f0-8efd-b7e1ca01614c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:32.881332 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec8619e-99dc-40f0-8efd-b7e1ca01614c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:32.881686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec8619e-99dc-40f0-8efd-b7e1ca01614c None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:32.882532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec8619e-99dc-40f0-8efd-b7e1ca01614c None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:32.882962 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec8619e-99dc-40f0-8efd-b7e1ca01614c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:32.884841 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec8619e-99dc-40f0-8efd-b7e1ca01614c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:32.894540 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-fa7bc817-3c1d-42fd-861b-08e4a2f630eb None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:32.898670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-fa7bc817-3c1d-42fd-861b-08e4a2f630eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:32.910326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 242/496] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Oct 16 00:44:32 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 381 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:32.920956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec8619e-99dc-40f0-8efd-b7e1ca01614c None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:32.932514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:33.049012 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 255/497] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:33.088518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec8619e-99dc-40f0-8efd-b7e1ca01614c None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:33.128836 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c917f854-a57b-4ee1-a096-409a54b702f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:33.129875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c917f854-a57b-4ee1-a096-409a54b702f5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:33.130340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c917f854-a57b-4ee1-a096-409a54b702f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:33.130793 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c917f854-a57b-4ee1-a096-409a54b702f5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:33.234326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c917f854-a57b-4ee1-a096-409a54b702f5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:33.236355 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c917f854-a57b-4ee1-a096-409a54b702f5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:33.248951 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:33.426998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 243/498] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:44:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 507 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 16 00:44:33.437663 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7bc817-3c1d-42fd-861b-08e4a2f630eb None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:33.486227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c917f854-a57b-4ee1-a096-409a54b702f5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:33.495802 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:33.649570 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7bc817-3c1d-42fd-861b-08e4a2f630eb None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:33.701811 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c917f854-a57b-4ee1-a096-409a54b702f5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:33.707769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419e6b11-69ba-4dbd-8835-b1854596d24f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:33.708485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419e6b11-69ba-4dbd-8835-b1854596d24f None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:33.708774 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419e6b11-69ba-4dbd-8835-b1854596d24f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:33.709048 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419e6b11-69ba-4dbd-8835-b1854596d24f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:33.710652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:33.738891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419e6b11-69ba-4dbd-8835-b1854596d24f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:33.740374 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419e6b11-69ba-4dbd-8835-b1854596d24f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:33.753284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c7d50a-3415-41a2-8309-4ee43f0475e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:33.753935 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c7d50a-3415-41a2-8309-4ee43f0475e4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:33.754211 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c7d50a-3415-41a2-8309-4ee43f0475e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:33.754488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c7d50a-3415-41a2-8309-4ee43f0475e4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:33.770671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-419e6b11-69ba-4dbd-8835-b1854596d24f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:33.770671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:33.770671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:33.770671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:33.770671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:33.770671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:33.770671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:33.770671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:33.770671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:33.770671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:33.770671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:33.770671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:33.770671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:33.770671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:33.770671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:33.770671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:33.771737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:33.771737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:33.771737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:33.771737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:33.771737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:33.771737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:33.771737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:33.771737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:33.771737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:33.771737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:33.771737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:33.771737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:33.771737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:33.771737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:33.771737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:33.771737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:33.771737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:33.772624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:33.772624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:33.772624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:33.772624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 256/499] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 336 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:33.783376 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c7d50a-3415-41a2-8309-4ee43f0475e4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:33.783912 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419e6b11-69ba-4dbd-8835-b1854596d24f None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:33.789195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-25c7d50a-3415-41a2-8309-4ee43f0475e4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:44:33.789195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:33.789195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:33.789195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:33.789195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:33.789195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:33.789195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:33.789195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:33.789195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:33.789195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:33.789195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:33.789195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:33.789195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:33.789195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:33.789195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:33.789195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:33.790767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:33.790767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:33.790767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:33.790767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:33.790767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:33.790767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:33.790767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:33.790767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:33.790767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:33.790767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:33.790767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:33.790767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:33.790767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:33.790767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:33.790767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:33.790767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:33.793253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:33.793253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:33.793253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:33.793253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:33.793253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:33.793253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:33.793253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:33.793253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:33.793253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:33.793253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:33.793253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:44:33.793253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:33.793253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 244/500] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:44:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 307 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:33.800685 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c7d50a-3415-41a2-8309-4ee43f0475e4 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:33.963535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:33.994585 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419e6b11-69ba-4dbd-8835-b1854596d24f None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:34.052876 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c7d50a-3415-41a2-8309-4ee43f0475e4 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:34.082941 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90eabad-a005-440e-9383-af766d8b76e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:34.084570 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90eabad-a005-440e-9383-af766d8b76e9 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:34.085160 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90eabad-a005-440e-9383-af766d8b76e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:34.085736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90eabad-a005-440e-9383-af766d8b76e9 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:34.087629 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90eabad-a005-440e-9383-af766d8b76e9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:34.088277 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90eabad-a005-440e-9383-af766d8b76e9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:34.091054 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90eabad-a005-440e-9383-af766d8b76e9 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:34.109326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 257/501] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:34.115200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90eabad-a005-440e-9383-af766d8b76e9 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:34.158892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344328dd-c752-4b4a-8cc8-a9dd6414f40d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:34.159715 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344328dd-c752-4b4a-8cc8-a9dd6414f40d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:34.159863 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344328dd-c752-4b4a-8cc8-a9dd6414f40d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:34.160076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344328dd-c752-4b4a-8cc8-a9dd6414f40d None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:34.160796 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344328dd-c752-4b4a-8cc8-a9dd6414f40d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:34.161105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344328dd-c752-4b4a-8cc8-a9dd6414f40d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:34.162560 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344328dd-c752-4b4a-8cc8-a9dd6414f40d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:34.189112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 245/502] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:44:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 389 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:34.198773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344328dd-c752-4b4a-8cc8-a9dd6414f40d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:34.294059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:34.325081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90eabad-a005-440e-9383-af766d8b76e9 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:34.405713 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:34.424578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded0bbce-a9d0-49b7-997a-85d6d327b3ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:34.425815 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded0bbce-a9d0-49b7-997a-85d6d327b3ab None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:34.426257 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded0bbce-a9d0-49b7-997a-85d6d327b3ab None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:34.426662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded0bbce-a9d0-49b7-997a-85d6d327b3ab None admin] PATH_INFO: `/v3/projects/service` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:34.435319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344328dd-c752-4b4a-8cc8-a9dd6414f40d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:34.448331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded0bbce-a9d0-49b7-997a-85d6d327b3ab None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:34.449829 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded0bbce-a9d0-49b7-997a-85d6d327b3ab None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:34.469734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-ded0bbce-a9d0-49b7-997a-85d6d327b3ab None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:34.469734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:34.469734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:34.469734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:34.469734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:34.469734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:34.469734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:34.469734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:34.469734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:34.469734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:34.469734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:34.469734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:34.469734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:44:34.469734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:44:34.469734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:44:34.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:44:34.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:34.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:34.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:34.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:34.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:34.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:34.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:34.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:34.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:34.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:34.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:34.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:34.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:34.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:34.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:34.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:34.472958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:44:34.472958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:44:34.472958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:44:34.472958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:44:34.472958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:44:34.472958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:44:34.472958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:34.472958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:34.472958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 258/503] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 357 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:34.477221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded0bbce-a9d0-49b7-997a-85d6d327b3ab None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:34.554129 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d7ab1d-d251-4608-ad1e-8b02f9482c7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:34.555145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d7ab1d-d251-4608-ad1e-8b02f9482c7c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:34.555731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d7ab1d-d251-4608-ad1e-8b02f9482c7c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:34.556224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d7ab1d-d251-4608-ad1e-8b02f9482c7c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:34.592557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded0bbce-a9d0-49b7-997a-85d6d327b3ab None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:34.596091 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d7ab1d-d251-4608-ad1e-8b02f9482c7c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:34.598929 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d7ab1d-d251-4608-ad1e-8b02f9482c7c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:34.630854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc49a16-b9f5-4f5c-825f-8749ab7a2caa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:34.631510 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc49a16-b9f5-4f5c-825f-8749ab7a2caa None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:34.631791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc49a16-b9f5-4f5c-825f-8749ab7a2caa None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:34.632035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc49a16-b9f5-4f5c-825f-8749ab7a2caa None admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:34.632753 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc49a16-b9f5-4f5c-825f-8749ab7a2caa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:34.633063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc49a16-b9f5-4f5c-825f-8749ab7a2caa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:34.634652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-86d7ab1d-d251-4608-ad1e-8b02f9482c7c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:34.634652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:34.634652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:34.634652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:34.634652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:34.634652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:34.634652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:34.634652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:34.634652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:34.634652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:34.634652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:34.634652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:34.634652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:34.634652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:34.634652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:34.634652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:34.638126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:34.638126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:34.638126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:34.638126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:34.638126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:34.638126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:34.638126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:34.638126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:34.638126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:34.638126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:34.638126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:34.638126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:34.638126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:34.638126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:34.638126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:34.638126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:34.638126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:34.641685 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:34.641685 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:34.641685 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:34.642322 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc49a16-b9f5-4f5c-825f-8749ab7a2caa None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:34.642549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 246/504] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 442 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:34.649013 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 259/505] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:44:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:34.650937 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d7ab1d-d251-4608-ad1e-8b02f9482c7c None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:34.655214 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc49a16-b9f5-4f5c-825f-8749ab7a2caa None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:34.713526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:34.804941 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc49a16-b9f5-4f5c-825f-8749ab7a2caa None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:34.845084 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87b27d6-1d82-473d-b9a1-d11f5337c568 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:34.845717 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87b27d6-1d82-473d-b9a1-d11f5337c568 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:34.846005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87b27d6-1d82-473d-b9a1-d11f5337c568 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:34.846286 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87b27d6-1d82-473d-b9a1-d11f5337c568 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:34.847070 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87b27d6-1d82-473d-b9a1-d11f5337c568 None admin] RBAC: Adding query filter params (role.id=bfe57c41a4da4b25b9b3e4761f220a46, scope.project.id=7deb2c3dae214df7895b40e7339bd6ea, user.id=de64d498da78448394b0d4d1851a2789) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:34.847386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87b27d6-1d82-473d-b9a1-d11f5337c568 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:34.848866 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87b27d6-1d82-473d-b9a1-d11f5337c568 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:34.856937 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 260/506] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:44:34 2022] GET /identity/v3/role_assignments?user.id=de64d498da78448394b0d4d1851a2789&scope.project.id=7deb2c3dae214df7895b40e7339bd6ea&role.id=bfe57c41a4da4b25b9b3e4761f220a46 => generated 608 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:34.896950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:34.928376 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d7ab1d-d251-4608-ad1e-8b02f9482c7c None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:34.965484 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e34766-2876-4c9a-995a-90f86b3ea179 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:34.965876 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e34766-2876-4c9a-995a-90f86b3ea179 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:34.966060 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e34766-2876-4c9a-995a-90f86b3ea179 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:34.967425 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 261/507] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:44:34.985575 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba76a1d-35b0-43af-acfb-1de49cdc0710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:34.986206 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba76a1d-35b0-43af-acfb-1de49cdc0710 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:34.986485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba76a1d-35b0-43af-acfb-1de49cdc0710 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:34.986758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba76a1d-35b0-43af-acfb-1de49cdc0710 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:34.987476 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba76a1d-35b0-43af-acfb-1de49cdc0710 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:34.987765 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba76a1d-35b0-43af-acfb-1de49cdc0710 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:34.989201 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba76a1d-35b0-43af-acfb-1de49cdc0710 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:34.994041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f639c56-29a7-4b35-84f6-e4e1149ded9a None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:34.994339 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f639c56-29a7-4b35-84f6-e4e1149ded9a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:34.994596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f639c56-29a7-4b35-84f6-e4e1149ded9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:35.005721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 247/508] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 356 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:35.014644 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba76a1d-35b0-43af-acfb-1de49cdc0710 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:35.024651 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-9f639c56-29a7-4b35-84f6-e4e1149ded9a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:35.027501 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-9f639c56-29a7-4b35-84f6-e4e1149ded9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:35.176171 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 262/509] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 16 00:44:35.259766 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba76a1d-35b0-43af-acfb-1de49cdc0710 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:35.345603 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702b7b31-eefd-490b-8e07-da6f30730c4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:35.346709 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702b7b31-eefd-490b-8e07-da6f30730c4d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:35.347310 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702b7b31-eefd-490b-8e07-da6f30730c4d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:35.347777 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702b7b31-eefd-490b-8e07-da6f30730c4d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:35.369059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702b7b31-eefd-490b-8e07-da6f30730c4d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:35.371612 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702b7b31-eefd-490b-8e07-da6f30730c4d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:35.380887 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:35.396824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-702b7b31-eefd-490b-8e07-da6f30730c4d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 16 00:44:35.396824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:35.396824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:35.396824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:35.396824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:35.396824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:35.396824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:35.396824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:35.396824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:35.396824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:35.396824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:35.396824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:35.396824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:44:35.396824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:44:35.396824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:44:35.396824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:35.397850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:35.399397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:35.399397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:35.399397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:44:35.399397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:44:35.399397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:44:35.399397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:44:35.399397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:44:35.399397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:44:35.399397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 16 00:44:35.399397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:35.399397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 248/510] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Oct 16 00:44:35 2022] GET /identity/v3/users/nova => generated 82 bytes in 384 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:35.412229 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f639c56-29a7-4b35-84f6-e4e1149ded9a None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:35.508106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702b7b31-eefd-490b-8e07-da6f30730c4d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:35.538947 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:35.603917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f639c56-29a7-4b35-84f6-e4e1149ded9a None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:35.646215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1d271c-0ada-4af0-8882-ba8ee58fa5a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:35.646887 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1d271c-0ada-4af0-8882-ba8ee58fa5a2 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:35.647176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1d271c-0ada-4af0-8882-ba8ee58fa5a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:35.647451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1d271c-0ada-4af0-8882-ba8ee58fa5a2 None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:35.648211 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc1d271c-0ada-4af0-8882-ba8ee58fa5a2 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:35.648523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc1d271c-0ada-4af0-8882-ba8ee58fa5a2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:35.650570 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc1d271c-0ada-4af0-8882-ba8ee58fa5a2 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:35.671960 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 263/511] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Oct 16 00:44:35 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:35.681096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1d271c-0ada-4af0-8882-ba8ee58fa5a2 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:35.809396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702b7b31-eefd-490b-8e07-da6f30730c4d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:35.849176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1d271c-0ada-4af0-8882-ba8ee58fa5a2 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:35.849176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:35.878082 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae9af6f-9fbe-4530-ae27-be143e9addae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:35.879435 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae9af6f-9fbe-4530-ae27-be143e9addae None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:35.879904 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae9af6f-9fbe-4530-ae27-be143e9addae None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:35.880359 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae9af6f-9fbe-4530-ae27-be143e9addae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:35.898970 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae9af6f-9fbe-4530-ae27-be143e9addae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:35.901946 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae9af6f-9fbe-4530-ae27-be143e9addae None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:35.904413 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fa5587-3ab2-4802-95b4-153d6cbe4653 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:35.905538 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fa5587-3ab2-4802-95b4-153d6cbe4653 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:35.906027 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fa5587-3ab2-4802-95b4-153d6cbe4653 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:35.906491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fa5587-3ab2-4802-95b4-153d6cbe4653 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:35.919986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fa5587-3ab2-4802-95b4-153d6cbe4653 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:35.922105 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fa5587-3ab2-4802-95b4-153d6cbe4653 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:35.932147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-4ae9af6f-9fbe-4530-ae27-be143e9addae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:35.932147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:35.932147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:35.932147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:35.932147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:35.932147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:35.932147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:35.932147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:35.932147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:35.932147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:35.932147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:35.932147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:35.932147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:35.932147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:35.932147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:35.932147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:35.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:35.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:35.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:35.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:35.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:35.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:35.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:35.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:35.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:35.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:35.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:35.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:35.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:35.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:35.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:35.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:35.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:35.936686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:35.936686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:35.936686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:35.939256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 249/512] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 431 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:35.941511 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b9768f-bb9f-450e-a9b8-28f433d7cc3a None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:35.942031 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b9768f-bb9f-450e-a9b8-28f433d7cc3a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:35.942398 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b9768f-bb9f-450e-a9b8-28f433d7cc3a None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:35.944729 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 250/513] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:35.946743 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b632da47-8552-4cd5-a0ef-c87162657e80 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:35.947278 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b632da47-8552-4cd5-a0ef-c87162657e80 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:35.947771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-97fa5587-3ab2-4802-95b4-153d6cbe4653 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:35.947771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:35.947771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:35.947771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:35.947771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:35.947771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:35.947771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:35.947771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:35.947771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:35.947771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:35.947771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:35.947771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:35.947771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:35.947771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:35.947771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:35.947771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:35.948732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:35.948732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:35.948732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:35.948732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:35.948732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:35.948732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:35.948732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:35.948732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:35.948732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:35.948732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:35.948732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:35.948732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:35.948732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:35.948732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:35.948732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:35.948732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:35.948732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:35.949911 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:35.949911 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:35.949911 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:35.950095 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b632da47-8552-4cd5-a0ef-c87162657e80 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:35.955435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 264/514] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:35.955435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fa5587-3ab2-4802-95b4-153d6cbe4653 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:35.955688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 251/515] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:35 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:35.955688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e16499-9606-432c-a8e7-96fa35736c6a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:35.955688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e16499-9606-432c-a8e7-96fa35736c6a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:35.955688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e16499-9606-432c-a8e7-96fa35736c6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:35.978636 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d9e16499-9606-432c-a8e7-96fa35736c6a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:35.979928 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d9e16499-9606-432c-a8e7-96fa35736c6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:36.037839 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:36.073153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 252/516] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:36.075283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e16499-9606-432c-a8e7-96fa35736c6a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:36.090560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fa5587-3ab2-4802-95b4-153d6cbe4653 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:36.130127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f757da-3c9f-491b-9771-568446cc6c48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:36.130766 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f757da-3c9f-491b-9771-568446cc6c48 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:36.131103 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f757da-3c9f-491b-9771-568446cc6c48 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:36.131346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f757da-3c9f-491b-9771-568446cc6c48 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:36.132019 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f757da-3c9f-491b-9771-568446cc6c48 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:36.132328 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f757da-3c9f-491b-9771-568446cc6c48 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:36.134545 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f757da-3c9f-491b-9771-568446cc6c48 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:36.155182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 265/517] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:36.156630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404839c3-06cc-4c9e-bf53-a2de46831930 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:36.156924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404839c3-06cc-4c9e-bf53-a2de46831930 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:36.157285 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404839c3-06cc-4c9e-bf53-a2de46831930 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:36.179210 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-404839c3-06cc-4c9e-bf53-a2de46831930 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:36.180519 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-404839c3-06cc-4c9e-bf53-a2de46831930 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:36.192895 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:36.269116 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 266/518] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:36.279879 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404839c3-06cc-4c9e-bf53-a2de46831930 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:36.319590 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e16499-9606-432c-a8e7-96fa35736c6a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:36.363666 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac9838a-ada8-47e4-8a9d-11ce89d0cfce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:36.364824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac9838a-ada8-47e4-8a9d-11ce89d0cfce None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:36.365118 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac9838a-ada8-47e4-8a9d-11ce89d0cfce None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:36.365386 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac9838a-ada8-47e4-8a9d-11ce89d0cfce None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:36.366109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac9838a-ada8-47e4-8a9d-11ce89d0cfce None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:36.366423 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac9838a-ada8-47e4-8a9d-11ce89d0cfce None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:36.368800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac9838a-ada8-47e4-8a9d-11ce89d0cfce None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:36.387208 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 253/519] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:36.389232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac9838a-ada8-47e4-8a9d-11ce89d0cfce None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:36.443570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:36.512865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac9838a-ada8-47e4-8a9d-11ce89d0cfce None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:36.553008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b90acfb-d3ca-4d24-b614-6ba8ac90ad65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:36.553808 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b90acfb-d3ca-4d24-b614-6ba8ac90ad65 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:36.554096 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b90acfb-d3ca-4d24-b614-6ba8ac90ad65 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:36.554339 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b90acfb-d3ca-4d24-b614-6ba8ac90ad65 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:36.562961 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b90acfb-d3ca-4d24-b614-6ba8ac90ad65 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:36.564634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b90acfb-d3ca-4d24-b614-6ba8ac90ad65 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:36.575440 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404839c3-06cc-4c9e-bf53-a2de46831930 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:36.577348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-4b90acfb-d3ca-4d24-b614-6ba8ac90ad65 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:36.577348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:36.577348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:36.577348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:36.577348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:36.577348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:36.577348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:36.577348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:36.577348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:36.577348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:36.577348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:36.577348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:36.577348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:44:36.577348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:44:36.577348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:44:36.579014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:44:36.579014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:36.579014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:36.579014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:36.579014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:36.579014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:36.579014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:36.579014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:36.579014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:36.579014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:36.579014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:36.579014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:36.579014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:36.579014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:36.579014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:36.579014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:36.579014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:36.580984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:44:36.580984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:44:36.580984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:44:36.580984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:44:36.580984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:44:36.580984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:44:36.580984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:36.580984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:36.583045 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 254/520] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:36.583960 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b90acfb-d3ca-4d24-b614-6ba8ac90ad65 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:36.628804 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e099a0-0dec-4129-bb03-bd6818e9260d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:36.629341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e099a0-0dec-4129-bb03-bd6818e9260d None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:36.629618 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e099a0-0dec-4129-bb03-bd6818e9260d None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:36.629887 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e099a0-0dec-4129-bb03-bd6818e9260d None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:36.630568 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e099a0-0dec-4129-bb03-bd6818e9260d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:36.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e099a0-0dec-4129-bb03-bd6818e9260d None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:36.652141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:36.663319 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 267/521] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:44:36 2022] POST /identity/v3/users => generated 254 bytes in 393 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:44:36.664616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e099a0-0dec-4129-bb03-bd6818e9260d None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:36.802153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b90acfb-d3ca-4d24-b614-6ba8ac90ad65 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:36.806751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:36.901356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e099a0-0dec-4129-bb03-bd6818e9260d None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:36.903469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86cfe045-9f15-415e-8762-1744e5797856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:36.904799 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cfe045-9f15-415e-8762-1744e5797856 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:36.905068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cfe045-9f15-415e-8762-1744e5797856 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:36.905547 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86cfe045-9f15-415e-8762-1744e5797856 None admin] PATH_INFO: `/v3/services/network` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:36.907226 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86cfe045-9f15-415e-8762-1744e5797856 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:36.912491 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86cfe045-9f15-415e-8762-1744e5797856 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:36.939568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-86cfe045-9f15-415e-8762-1744e5797856 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 16 00:44:36.939568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:36.939568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:44:36.939568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:44:36.939568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:44:36.939568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:44:36.939568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:44:36.939568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:36.939568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 16 00:44:36.939568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:36.939568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:44:36.939568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:36.939568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:36.939568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:36.939568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:36.939568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:36.939568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:36.940706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:36.940706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:36.940706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:36.940706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:36.940706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:36.940706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:36.940706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:44:36.940706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:44:36.940706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:44:36.940706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:44:36.940706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:36.940706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:36.940706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:36.940706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:36.940706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:36.940706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:36.941570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:36.941570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:36.941570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:36.941570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:36.941570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:36.941570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:36.941570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:36.941570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:36.941570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:36.941570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:36.941570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:44:36.941570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:36.941570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 16 00:44:36.941570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:36.942322 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 255/522] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:36 2022] GET /identity/v3/services/network => generated 88 bytes in 358 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:36.943652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86cfe045-9f15-415e-8762-1744e5797856 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:36.959768 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a22ab48-ec50-448e-b97b-eeed1f934f1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:36.960838 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a22ab48-ec50-448e-b97b-eeed1f934f1b None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:36.961313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a22ab48-ec50-448e-b97b-eeed1f934f1b None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:36.961852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a22ab48-ec50-448e-b97b-eeed1f934f1b None admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:36.965050 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a22ab48-ec50-448e-b97b-eeed1f934f1b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:36.965050 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a22ab48-ec50-448e-b97b-eeed1f934f1b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:36.966510 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a22ab48-ec50-448e-b97b-eeed1f934f1b None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:36.991158 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 268/523] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:44:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:36.992803 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a22ab48-ec50-448e-b97b-eeed1f934f1b None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:37.125992 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:37.184673 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86cfe045-9f15-415e-8762-1744e5797856 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:37.209445 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a22ab48-ec50-448e-b97b-eeed1f934f1b None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:37.242205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985daed8-5fc9-4593-9673-4a20b2d3cbb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:37.243469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985daed8-5fc9-4593-9673-4a20b2d3cbb8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:37.243756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985daed8-5fc9-4593-9673-4a20b2d3cbb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:37.244031 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985daed8-5fc9-4593-9673-4a20b2d3cbb8 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:37.244627 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985daed8-5fc9-4593-9673-4a20b2d3cbb8 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:37.246359 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985daed8-5fc9-4593-9673-4a20b2d3cbb8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:37.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-985daed8-5fc9-4593-9673-4a20b2d3cbb8 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 16 00:44:37.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:37.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:44:37.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:44:37.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:44:37.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:44:37.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:44:37.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:37.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 16 00:44:37.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:37.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:44:37.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:37.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:37.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:37.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:37.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:37.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:37.259665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:37.259665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:37.259665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:37.259665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:37.259665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:37.259665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:37.259665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:44:37.259665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:44:37.259665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:44:37.259665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:44:37.259665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:37.259665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:37.259665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:37.259665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:37.259665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:37.259665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:37.261365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:37.261365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:37.261365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:37.261365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:37.261365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:37.261365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:37.261365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:37.261365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:37.261365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:37.261365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:37.261365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:44:37.261365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:37.261365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 16 00:44:37.261365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:37.261365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 256/524] 10.222.0.25 () {56 vars in 1129 bytes} [Sun Oct 16 00:44:36 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:37.263910 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985daed8-5fc9-4593-9673-4a20b2d3cbb8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:37.287955 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7898f578-0fbf-464b-8831-7e9604d390b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:37.289328 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7898f578-0fbf-464b-8831-7e9604d390b9 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:37.290241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7898f578-0fbf-464b-8831-7e9604d390b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:37.291030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7898f578-0fbf-464b-8831-7e9604d390b9 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:37.292519 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7898f578-0fbf-464b-8831-7e9604d390b9 None admin] RBAC: Adding query filter params (name=network) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:37.293558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7898f578-0fbf-464b-8831-7e9604d390b9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:37.296559 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7898f578-0fbf-464b-8831-7e9604d390b9 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:37.300529 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:37.309476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 269/525] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 16 00:44:36 2022] GET /identity/v3/services?name=network => generated 124 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:37.310691 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7898f578-0fbf-464b-8831-7e9604d390b9 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:37.437546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985daed8-5fc9-4593-9673-4a20b2d3cbb8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:37.463722 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7898f578-0fbf-464b-8831-7e9604d390b9 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:37.496703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:37.509208 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38539665-2ef1-4da6-9b85-4b3f524b0b6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:37.510612 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38539665-2ef1-4da6-9b85-4b3f524b0b6a None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:37.510806 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504e2695-233e-4716-8eff-f12704df52f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:37.511355 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504e2695-233e-4716-8eff-f12704df52f4 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:37.511641 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504e2695-233e-4716-8eff-f12704df52f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:37.511915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504e2695-233e-4716-8eff-f12704df52f4 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:37.512146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38539665-2ef1-4da6-9b85-4b3f524b0b6a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:37.512563 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504e2695-233e-4716-8eff-f12704df52f4 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:37.512872 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504e2695-233e-4716-8eff-f12704df52f4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:37.513260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38539665-2ef1-4da6-9b85-4b3f524b0b6a None admin] PATH_INFO: `/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/4eba26813e154595a12a67fe4be5eeff/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:37.514264 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504e2695-233e-4716-8eff-f12704df52f4 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:37.517243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38539665-2ef1-4da6-9b85-4b3f524b0b6a None admin] RBAC: Authorizing `identity:create_grant(project_id=7deb2c3dae214df7895b40e7339bd6ea, user_id=4eba26813e154595a12a67fe4be5eeff, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:37.519583 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 270/526] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Oct 16 00:44:37 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:37.520433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504e2695-233e-4716-8eff-f12704df52f4 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:37.520737 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38539665-2ef1-4da6-9b85-4b3f524b0b6a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:37.546777 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 257/527] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:44:37 2022] PUT /identity/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/4eba26813e154595a12a67fe4be5eeff/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:44:37.549325 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38539665-2ef1-4da6-9b85-4b3f524b0b6a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:37.672415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504e2695-233e-4716-8eff-f12704df52f4 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:37.715013 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee6d1ec-7a63-4d9c-9f42-691fdb16dcee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:37.715622 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee6d1ec-7a63-4d9c-9f42-691fdb16dcee None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:37.715907 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee6d1ec-7a63-4d9c-9f42-691fdb16dcee None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:37.716296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee6d1ec-7a63-4d9c-9f42-691fdb16dcee None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:37.716842 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee6d1ec-7a63-4d9c-9f42-691fdb16dcee None admin] RBAC: Adding query filter params (type=network) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:37.717150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee6d1ec-7a63-4d9c-9f42-691fdb16dcee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:37.718656 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee6d1ec-7a63-4d9c-9f42-691fdb16dcee None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:37.725638 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 271/528] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 16 00:44:37 2022] GET /identity/v3/services?type=network => generated 124 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:37.728119 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c14b29d-1c7e-4739-bd5e-c38f183023db None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:37.728530 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c14b29d-1c7e-4739-bd5e-c38f183023db None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:37.728825 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c14b29d-1c7e-4739-bd5e-c38f183023db None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:37.735474 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 272/529] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:37 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:37.736889 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01a14e7-0cb6-4524-99f8-efd1e8aa7cc7 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:37.737180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01a14e7-0cb6-4524-99f8-efd1e8aa7cc7 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:37.737505 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01a14e7-0cb6-4524-99f8-efd1e8aa7cc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:37.768646 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-e01a14e7-0cb6-4524-99f8-efd1e8aa7cc7 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:37.770894 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-e01a14e7-0cb6-4524-99f8-efd1e8aa7cc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:37.774999 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:37.798796 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38539665-2ef1-4da6-9b85-4b3f524b0b6a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:37.843693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4744f242-9807-45e0-ac01-a4d17643ee17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:37.844774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4744f242-9807-45e0-ac01-a4d17643ee17 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:37.845284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4744f242-9807-45e0-ac01-a4d17643ee17 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:37.845826 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4744f242-9807-45e0-ac01-a4d17643ee17 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:37.846830 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4744f242-9807-45e0-ac01-a4d17643ee17 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:37.847422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4744f242-9807-45e0-ac01-a4d17643ee17 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:37.849526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4744f242-9807-45e0-ac01-a4d17643ee17 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:37.856994 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 258/530] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Oct 16 00:44:37 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:37.873530 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:37.917974 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 273/531] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:38.440109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4744f242-9807-45e0-ac01-a4d17643ee17 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:38.581928 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4744f242-9807-45e0-ac01-a4d17643ee17 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:38.630624 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5393809b-d6f9-44fa-8f62-d9e1d74601b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:38.631684 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5393809b-d6f9-44fa-8f62-d9e1d74601b0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:38.632177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5393809b-d6f9-44fa-8f62-d9e1d74601b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:38.632667 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5393809b-d6f9-44fa-8f62-d9e1d74601b0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:38.649343 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5393809b-d6f9-44fa-8f62-d9e1d74601b0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:38.655099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-5393809b-d6f9-44fa-8f62-d9e1d74601b0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:44:38.655099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:38.655099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:38.655099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:38.655099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:38.655099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:38.655099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:38.655099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:38.655099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:38.655099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:38.655099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:38.655099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:38.655099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:38.655099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:38.655099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:38.655099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:38.656090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:38.656090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:38.656090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:38.656090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:38.656090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:38.656090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:38.656090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:38.656090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:38.656090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:38.656090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:38.656090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:38.656090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:38.656090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:38.656090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:38.656090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:38.656090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:38.656956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:38.656956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:38.656956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:38.656956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:38.656956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:38.656956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:38.656956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:38.656956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:38.656956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:38.656956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:38.656956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:44:38.656956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:38.659973 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 259/532] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:44:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:38.669750 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01a14e7-0cb6-4524-99f8-efd1e8aa7cc7 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:38.882595 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01a14e7-0cb6-4524-99f8-efd1e8aa7cc7 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:38.948842 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491b552c-f7ff-46d7-a713-65579e7dec97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:38.949892 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491b552c-f7ff-46d7-a713-65579e7dec97 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:38.950370 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491b552c-f7ff-46d7-a713-65579e7dec97 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:38.950847 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491b552c-f7ff-46d7-a713-65579e7dec97 None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:38.952128 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491b552c-f7ff-46d7-a713-65579e7dec97 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:38.952658 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491b552c-f7ff-46d7-a713-65579e7dec97 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:38.955662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491b552c-f7ff-46d7-a713-65579e7dec97 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:38.973531 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 274/533] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:44:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:38.983499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5393809b-d6f9-44fa-8f62-d9e1d74601b0 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:39.013758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:39.191017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5393809b-d6f9-44fa-8f62-d9e1d74601b0 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:39.267835 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac65a51d-f6ed-4c2c-acc5-2127b21223be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:39.268912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac65a51d-f6ed-4c2c-acc5-2127b21223be None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:39.269373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac65a51d-f6ed-4c2c-acc5-2127b21223be None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:39.269845 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac65a51d-f6ed-4c2c-acc5-2127b21223be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:39.291421 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac65a51d-f6ed-4c2c-acc5-2127b21223be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:39.292933 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac65a51d-f6ed-4c2c-acc5-2127b21223be None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:39.317587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-ac65a51d-f6ed-4c2c-acc5-2127b21223be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:39.317587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:39.317587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:39.317587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:39.317587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:39.317587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:39.317587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:39.317587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:39.317587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:39.317587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:39.317587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:39.317587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:39.317587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:39.317587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:39.317587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:39.317587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:39.318626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:39.318626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:39.318626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:39.318626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:39.318626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:39.318626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:39.318626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:39.318626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:39.318626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:39.318626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:39.318626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:39.318626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:39.318626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:39.318626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:39.318626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:39.318626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:39.318626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:39.319548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:39.319548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:39.319548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:39.319548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 260/534] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:39.328581 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491b552c-f7ff-46d7-a713-65579e7dec97 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:39.446612 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:39.536675 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491b552c-f7ff-46d7-a713-65579e7dec97 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:39.592014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3115edcf-19c3-4340-89fe-b3c78d211f70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:39.592661 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3115edcf-19c3-4340-89fe-b3c78d211f70 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:39.592949 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3115edcf-19c3-4340-89fe-b3c78d211f70 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:39.593229 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3115edcf-19c3-4340-89fe-b3c78d211f70 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:39.593901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3115edcf-19c3-4340-89fe-b3c78d211f70 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:39.594218 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3115edcf-19c3-4340-89fe-b3c78d211f70 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:39.595685 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3115edcf-19c3-4340-89fe-b3c78d211f70 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:39.616313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 275/535] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:39.625797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac65a51d-f6ed-4c2c-acc5-2127b21223be None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:39.716460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367c41d5-7080-4bfa-8c4e-92e11e6a4942 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:39.716670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367c41d5-7080-4bfa-8c4e-92e11e6a4942 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:39.716967 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367c41d5-7080-4bfa-8c4e-92e11e6a4942 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:39.718542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 276/536] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:44:39.727756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6366713a-7e84-48fc-bbae-1260a6551d8c None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:39.728173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6366713a-7e84-48fc-bbae-1260a6551d8c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:39.728629 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6366713a-7e84-48fc-bbae-1260a6551d8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:39.743450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:39.772997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-6366713a-7e84-48fc-bbae-1260a6551d8c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:39.774432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6366713a-7e84-48fc-bbae-1260a6551d8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:39.835180 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac65a51d-f6ed-4c2c-acc5-2127b21223be None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:39.918411 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017f6f2c-5c68-4839-9560-92e37384abb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:39.919549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017f6f2c-5c68-4839-9560-92e37384abb0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:39.920050 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017f6f2c-5c68-4839-9560-92e37384abb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:39.920547 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017f6f2c-5c68-4839-9560-92e37384abb0 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:39.938434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017f6f2c-5c68-4839-9560-92e37384abb0 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:39.941862 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017f6f2c-5c68-4839-9560-92e37384abb0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:39.958588 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 277/537] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:39.973104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-017f6f2c-5c68-4839-9560-92e37384abb0 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 16 00:44:39.973104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:39.973104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:39.973104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:39.973104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:39.973104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:39.973104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:39.973104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:39.973104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:39.973104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:39.973104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:39.973104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:39.973104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:44:39.973104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:44:39.973104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:44:39.973104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:39.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:39.977488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:39.977488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:39.977488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:44:39.977488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:44:39.977488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:44:39.977488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:44:39.977488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:44:39.977488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:44:39.977488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 16 00:44:39.977488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:39.977488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 261/538] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:44:39 2022] GET /identity/v3/users/swift => generated 83 bytes in 350 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:39.988244 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6366713a-7e84-48fc-bbae-1260a6551d8c None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:40.053043 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:40.158544 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6366713a-7e84-48fc-bbae-1260a6551d8c None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:40.210885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d24a0a-aefa-45f1-80ee-81b17673e069 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:40.212050 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d24a0a-aefa-45f1-80ee-81b17673e069 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:40.212336 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d24a0a-aefa-45f1-80ee-81b17673e069 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:40.212579 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d24a0a-aefa-45f1-80ee-81b17673e069 None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:40.213324 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d24a0a-aefa-45f1-80ee-81b17673e069 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:40.215489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d24a0a-aefa-45f1-80ee-81b17673e069 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:40.215489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d24a0a-aefa-45f1-80ee-81b17673e069 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:40.249526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 278/539] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Oct 16 00:44:39 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:40.258894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017f6f2c-5c68-4839-9560-92e37384abb0 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:40.423045 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017f6f2c-5c68-4839-9560-92e37384abb0 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:40.467397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:40.480857 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875d33ed-55ee-4c94-af1f-7853f69031a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:40.481510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875d33ed-55ee-4c94-af1f-7853f69031a3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:40.481794 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875d33ed-55ee-4c94-af1f-7853f69031a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:40.482066 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875d33ed-55ee-4c94-af1f-7853f69031a3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:40.503241 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875d33ed-55ee-4c94-af1f-7853f69031a3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:40.504658 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875d33ed-55ee-4c94-af1f-7853f69031a3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:40.521365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-875d33ed-55ee-4c94-af1f-7853f69031a3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:40.521365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:40.521365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:40.521365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:40.521365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:40.521365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:40.521365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:40.521365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:40.521365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:40.521365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:40.521365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:40.521365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:40.521365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:40.521365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:40.521365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:40.521365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:40.522533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:40.522533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:40.522533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:40.522533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:40.522533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:40.522533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:40.522533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:40.522533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:40.522533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:40.522533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:40.522533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:40.522533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:40.522533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:40.522533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:40.522533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:40.522533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:40.522533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:40.523645 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:40.523645 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:40.523645 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:40.523645 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 262/540] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:40.532064 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d24a0a-aefa-45f1-80ee-81b17673e069 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:40.542533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875d33ed-55ee-4c94-af1f-7853f69031a3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:40.718400 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875d33ed-55ee-4c94-af1f-7853f69031a3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:40.762833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbad380-18d6-47c9-84be-3e5bec0f7fbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:40.764484 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbad380-18d6-47c9-84be-3e5bec0f7fbb None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:40.764484 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbad380-18d6-47c9-84be-3e5bec0f7fbb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:40.764484 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbad380-18d6-47c9-84be-3e5bec0f7fbb None admin] PATH_INFO: `/v3/roles/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:40.788893 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbad380-18d6-47c9-84be-3e5bec0f7fbb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:40.796593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-8fbad380-18d6-47c9-84be-3e5bec0f7fbb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:40.796593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:40.796593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:40.796593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:40.796593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:40.796593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:40.796593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:40.796593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:40.796593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:40.796593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:40.796593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:40.796593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:40.796593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:40.796593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:40.796593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:40.796593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:40.798429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:40.798429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:40.798429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:40.798429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:40.798429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:40.798429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:40.798429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:40.798429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:40.798429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:40.798429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:40.798429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:40.798429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:40.798429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:40.798429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:40.798429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:40.798429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:40.802022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:40.802022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:40.802022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:40.802022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:40.802022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:40.802022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:40.802022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:40.802022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:40.802022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:40.802022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:40.802022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:40.802022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:40.802022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 263/541] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 16 00:44:40 2022] GET /identity/v3/roles/service => generated 85 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:40.804830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:40.810891 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbad380-18d6-47c9-84be-3e5bec0f7fbb None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:40.819146 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d24a0a-aefa-45f1-80ee-81b17673e069 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:40.867092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:40.872535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b922923f-82fc-413e-a5a7-430cebbd6a8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:40.874135 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b922923f-82fc-413e-a5a7-430cebbd6a8f None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:40.875141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b922923f-82fc-413e-a5a7-430cebbd6a8f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:40.875776 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b922923f-82fc-413e-a5a7-430cebbd6a8f None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:40.877202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b922923f-82fc-413e-a5a7-430cebbd6a8f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:40.877915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b922923f-82fc-413e-a5a7-430cebbd6a8f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:40.881288 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b922923f-82fc-413e-a5a7-430cebbd6a8f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:40.902215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 279/542] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 370 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:40.905133 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd56fdf0-4cd7-49c8-a7b5-3364c6f114f8 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:40.905615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd56fdf0-4cd7-49c8-a7b5-3364c6f114f8 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:40.906123 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd56fdf0-4cd7-49c8-a7b5-3364c6f114f8 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:40.908041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 280/543] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:40.910687 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd56fdf0-4cd7-49c8-a7b5-3364c6f114f8 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:40.949814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbad380-18d6-47c9-84be-3e5bec0f7fbb None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:40.995589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8715ef72-168d-4143-bdca-a751c72b8b17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:40.996341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8715ef72-168d-4143-bdca-a751c72b8b17 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:40.996812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8715ef72-168d-4143-bdca-a751c72b8b17 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:40.997163 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8715ef72-168d-4143-bdca-a751c72b8b17 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:40.998038 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8715ef72-168d-4143-bdca-a751c72b8b17 None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:40.998429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8715ef72-168d-4143-bdca-a751c72b8b17 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:41.000104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8715ef72-168d-4143-bdca-a751c72b8b17 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:41.016439 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 264/544] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 16 00:44:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:41.016439 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c89533-e5be-4464-a473-1eb772e8a6f3 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:41.016439 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c89533-e5be-4464-a473-1eb772e8a6f3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:41.016439 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c89533-e5be-4464-a473-1eb772e8a6f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:41.042074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-52c89533-e5be-4464-a473-1eb772e8a6f3 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:41.044784 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-52c89533-e5be-4464-a473-1eb772e8a6f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:41.059915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd56fdf0-4cd7-49c8-a7b5-3364c6f114f8 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:41.113965 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6316c7-3bd9-485b-be0f-bdba2b11c76d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:41.114981 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6316c7-3bd9-485b-be0f-bdba2b11c76d None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:41.115465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6316c7-3bd9-485b-be0f-bdba2b11c76d None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:41.116021 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6316c7-3bd9-485b-be0f-bdba2b11c76d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:41.127232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6316c7-3bd9-485b-be0f-bdba2b11c76d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:41.129827 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6316c7-3bd9-485b-be0f-bdba2b11c76d None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:41.153598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-ab6316c7-3bd9-485b-be0f-bdba2b11c76d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:41.153598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:41.153598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:41.153598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:41.153598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:41.153598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:41.153598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:41.153598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:41.153598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:41.153598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:41.153598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:41.153598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:41.153598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:44:41.153598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:44:41.153598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:44:41.154796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:44:41.154796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:41.154796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:41.154796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:41.154796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:41.154796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:41.154796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:41.154796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:41.154796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:41.154796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:41.154796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:41.154796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:41.154796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:41.154796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:41.154796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:41.154796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:41.154796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:41.156016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:44:41.156016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:44:41.156016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:44:41.156016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:44:41.156016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:44:41.156016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:44:41.156016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:41.156016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:41.158642 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 281/545] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:41.160588 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6316c7-3bd9-485b-be0f-bdba2b11c76d None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:41.191001 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 265/546] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:41.191001 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c89533-e5be-4464-a473-1eb772e8a6f3 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:41.203017 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:41.270562 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:41.370125 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6316c7-3bd9-485b-be0f-bdba2b11c76d None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:41.376961 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c89533-e5be-4464-a473-1eb772e8a6f3 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:41.419540 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802e8961-06d3-4a02-9530-f8793159a23a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:41.420395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802e8961-06d3-4a02-9530-f8793159a23a None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:41.420859 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802e8961-06d3-4a02-9530-f8793159a23a None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:41.421318 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802e8961-06d3-4a02-9530-f8793159a23a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:41.436542 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a736ca3-9d86-4d35-b15b-acb5c2bccf2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:41.438211 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a736ca3-9d86-4d35-b15b-acb5c2bccf2c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:41.438676 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a736ca3-9d86-4d35-b15b-acb5c2bccf2c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:41.439193 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a736ca3-9d86-4d35-b15b-acb5c2bccf2c None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:41.440778 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a736ca3-9d86-4d35-b15b-acb5c2bccf2c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:41.441240 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802e8961-06d3-4a02-9530-f8793159a23a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:41.441342 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a736ca3-9d86-4d35-b15b-acb5c2bccf2c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:41.443741 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802e8961-06d3-4a02-9530-f8793159a23a None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:41.443874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a736ca3-9d86-4d35-b15b-acb5c2bccf2c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:41.474927 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 266/547] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:44:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:41.475185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-802e8961-06d3-4a02-9530-f8793159a23a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:41.475185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:41.475185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:41.475185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:41.475185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:41.475185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:41.475185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:41.475185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:41.475185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:41.475185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:41.475185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:41.475185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:41.475185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:41.475185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:41.475185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:41.475185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:41.477401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:41.477401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:41.477401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:41.477401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:41.477401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:41.477401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:41.477401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:41.477401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:41.477401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:41.477401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:41.477401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:41.477401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:41.477401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:41.477401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:41.477401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:41.477401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:41.477401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:41.484062 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:41.484062 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:41.484062 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:41.484407 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e542b8b-9db9-46f9-8d69-0a500865d58e None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:41.484407 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e542b8b-9db9-46f9-8d69-0a500865d58e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:41.484407 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e542b8b-9db9-46f9-8d69-0a500865d58e None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:41.484407 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 267/548] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:41 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:41.484407 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f9f48d-7fff-4031-867a-13ec827b3d3f None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:41.484407 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f9f48d-7fff-4031-867a-13ec827b3d3f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:41.485111 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f9f48d-7fff-4031-867a-13ec827b3d3f None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:41.485237 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 282/549] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 321 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:41.485362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 268/550] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:41 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:41.485893 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802e8961-06d3-4a02-9530-f8793159a23a None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:41.487058 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f9f48d-7fff-4031-867a-13ec827b3d3f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:41.580605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:41.609843 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:41.622696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f9f48d-7fff-4031-867a-13ec827b3d3f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:41.655044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802e8961-06d3-4a02-9530-f8793159a23a None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:41.666250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f00a08e-7ee0-4d98-b296-d269528e4a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:41.667114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f00a08e-7ee0-4d98-b296-d269528e4a57 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:41.667482 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f00a08e-7ee0-4d98-b296-d269528e4a57 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:41.667867 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f00a08e-7ee0-4d98-b296-d269528e4a57 None admin] PATH_INFO: `/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/de64d498da78448394b0d4d1851a2789/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:41.670331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f00a08e-7ee0-4d98-b296-d269528e4a57 None admin] RBAC: Authorizing `identity:create_grant(project_id=7deb2c3dae214df7895b40e7339bd6ea, user_id=de64d498da78448394b0d4d1851a2789, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:41.671939 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f00a08e-7ee0-4d98-b296-d269528e4a57 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:41.688428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 269/551] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:44:41 2022] PUT /identity/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/de64d498da78448394b0d4d1851a2789/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:44:41.690242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f00a08e-7ee0-4d98-b296-d269528e4a57 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:41.723989 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d59b970-ca99-4b16-a81a-53a2ba33b283 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:41.724787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d59b970-ca99-4b16-a81a-53a2ba33b283 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:41.725265 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d59b970-ca99-4b16-a81a-53a2ba33b283 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:41.725420 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d59b970-ca99-4b16-a81a-53a2ba33b283 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:41.726254 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d59b970-ca99-4b16-a81a-53a2ba33b283 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:41.726652 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d59b970-ca99-4b16-a81a-53a2ba33b283 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:41.728392 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d59b970-ca99-4b16-a81a-53a2ba33b283 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:41.751756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 283/552] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:41.751756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b59e4c-ecfc-4540-93f2-d46a29aaea6a None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:41.751756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b59e4c-ecfc-4540-93f2-d46a29aaea6a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:41.751756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b59e4c-ecfc-4540-93f2-d46a29aaea6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:41.771749 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-d7b59e4c-ecfc-4540-93f2-d46a29aaea6a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:41.773010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d7b59e4c-ecfc-4540-93f2-d46a29aaea6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:41.881940 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 284/553] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:41.883407 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a966492a-9737-4a17-aa34-ad7c66de80a3 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:41.883757 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a966492a-9737-4a17-aa34-ad7c66de80a3 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:41.884033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a966492a-9737-4a17-aa34-ad7c66de80a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:41.901141 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f00a08e-7ee0-4d98-b296-d269528e4a57 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:41.904988 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-a966492a-9737-4a17-aa34-ad7c66de80a3 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:41.906272 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-a966492a-9737-4a17-aa34-ad7c66de80a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:41.963733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa76c89b-8b29-4be0-add0-cc4ec9121f83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:41.965186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa76c89b-8b29-4be0-add0-cc4ec9121f83 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:41.965904 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa76c89b-8b29-4be0-add0-cc4ec9121f83 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:41.966572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa76c89b-8b29-4be0-add0-cc4ec9121f83 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:41.967614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa76c89b-8b29-4be0-add0-cc4ec9121f83 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:41.969616 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa76c89b-8b29-4be0-add0-cc4ec9121f83 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:41.978186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:41.999314 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 270/554] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 16 00:44:41 2022] POST /identity/v3/services => generated 240 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:44:42.001443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa76c89b-8b29-4be0-add0-cc4ec9121f83 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:42.027626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:42.039433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 285/555] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:41 2022] POST /identity/v3/auth/tokens => generated 898 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 16 00:44:42.211569 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa76c89b-8b29-4be0-add0-cc4ec9121f83 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:42.273144 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d977f0-606c-4b08-87e3-2c812dd111e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:42.274217 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d977f0-606c-4b08-87e3-2c812dd111e4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:42.274651 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d977f0-606c-4b08-87e3-2c812dd111e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:42.275173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d977f0-606c-4b08-87e3-2c812dd111e4 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:42.292192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d977f0-606c-4b08-87e3-2c812dd111e4 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:42.295118 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d977f0-606c-4b08-87e3-2c812dd111e4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:42.313219 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a966492a-9737-4a17-aa34-ad7c66de80a3 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:42.316510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-e6d977f0-606c-4b08-87e3-2c812dd111e4 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 16 00:44:42.316510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:42.316510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:42.316510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:42.316510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:42.316510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:42.316510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:42.316510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:42.316510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:42.316510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:42.316510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:42.316510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:42.316510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:44:42.316510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:44:42.316510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:44:42.316510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:42.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:42.318510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:42.318510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:42.318510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:44:42.318510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:44:42.318510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:44:42.318510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:44:42.318510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:44:42.318510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:44:42.318510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 16 00:44:42.318510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:42.319123 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 271/556] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 16 00:44:41 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:42.327364 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d977f0-606c-4b08-87e3-2c812dd111e4 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:42.468959 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a966492a-9737-4a17-aa34-ad7c66de80a3 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:42.487259 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:42.515976 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c168c0-837f-492f-ae2b-0c977fd75764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:42.517027 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c168c0-837f-492f-ae2b-0c977fd75764 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:42.517496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c168c0-837f-492f-ae2b-0c977fd75764 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:42.517982 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c168c0-837f-492f-ae2b-0c977fd75764 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:42.519223 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c168c0-837f-492f-ae2b-0c977fd75764 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:42.521864 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c168c0-837f-492f-ae2b-0c977fd75764 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:42.526645 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d977f0-606c-4b08-87e3-2c812dd111e4 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:42.553418 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 286/557] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 16 00:44:42 2022] POST /identity/v3/services => generated 251 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:44:42.555695 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c168c0-837f-492f-ae2b-0c977fd75764 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:42.578895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa919427-4c26-4b3b-89cf-11c13c067ef3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:42.580378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa919427-4c26-4b3b-89cf-11c13c067ef3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:42.580923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa919427-4c26-4b3b-89cf-11c13c067ef3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:42.581504 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa919427-4c26-4b3b-89cf-11c13c067ef3 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:42.582580 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa919427-4c26-4b3b-89cf-11c13c067ef3 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:42.583151 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa919427-4c26-4b3b-89cf-11c13c067ef3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:42.585104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa919427-4c26-4b3b-89cf-11c13c067ef3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:42.608920 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 272/558] 10.222.0.25 () {56 vars in 1149 bytes} [Sun Oct 16 00:44:42 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:42.619064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa919427-4c26-4b3b-89cf-11c13c067ef3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:42.679575 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:42.834473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c168c0-837f-492f-ae2b-0c977fd75764 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:42.881068 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7043ac3-4659-4e4a-a824-da8339b5d1d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:42.881736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7043ac3-4659-4e4a-a824-da8339b5d1d0 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:42.882017 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7043ac3-4659-4e4a-a824-da8339b5d1d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:42.882292 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7043ac3-4659-4e4a-a824-da8339b5d1d0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:42.900216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7043ac3-4659-4e4a-a824-da8339b5d1d0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:42.905734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-c7043ac3-4659-4e4a-a824-da8339b5d1d0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:44:42.905734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:42.905734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:42.905734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:42.905734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:42.905734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:42.905734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:42.905734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:42.905734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:42.905734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:42.905734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:42.905734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:42.905734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:42.905734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:42.905734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:42.905734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:42.906748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:42.906748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:42.906748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:42.906748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:42.906748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:42.906748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:42.906748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:42.906748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:42.906748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:42.906748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:42.906748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:42.906748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:42.906748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:42.906748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:42.906748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:42.906748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:42.908088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:42.908088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:42.908088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:42.908088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:42.908088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:42.908088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:42.908088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:42.908088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:42.908088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:42.908088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:42.908088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:44:42.908088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:42.908088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 287/559] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:44:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 353 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:42.911691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa919427-4c26-4b3b-89cf-11c13c067ef3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:42.914196 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7043ac3-4659-4e4a-a824-da8339b5d1d0 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:42.961791 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:42.983333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81247c7-54b9-4a8e-811f-460b87be2bc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:42.984418 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81247c7-54b9-4a8e-811f-460b87be2bc0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:42.984984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81247c7-54b9-4a8e-811f-460b87be2bc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:42.985459 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81247c7-54b9-4a8e-811f-460b87be2bc0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:43.005156 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81247c7-54b9-4a8e-811f-460b87be2bc0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:43.007972 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81247c7-54b9-4a8e-811f-460b87be2bc0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:43.032215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7043ac3-4659-4e4a-a824-da8339b5d1d0 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:43.035413 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:43.037559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-d81247c7-54b9-4a8e-811f-460b87be2bc0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:43.037559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:43.037559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:43.037559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:43.037559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:43.037559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:43.037559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:43.037559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:43.037559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:43.037559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:43.037559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:43.037559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:43.037559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:43.037559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:43.037559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:43.037559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:43.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:43.041149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:43.041149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:43.041149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:43.041149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 273/560] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 423 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:43.052522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81247c7-54b9-4a8e-811f-460b87be2bc0 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:43.081008 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fd4e49-5769-4418-ae17-d4896fe6c8dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:43.081641 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fd4e49-5769-4418-ae17-d4896fe6c8dc None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:43.081920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fd4e49-5769-4418-ae17-d4896fe6c8dc None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:43.082193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fd4e49-5769-4418-ae17-d4896fe6c8dc None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:43.082896 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fd4e49-5769-4418-ae17-d4896fe6c8dc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:43.083216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fd4e49-5769-4418-ae17-d4896fe6c8dc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:43.084808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fd4e49-5769-4418-ae17-d4896fe6c8dc None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:43.096457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 288/561] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:44:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:43.102938 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fd4e49-5769-4418-ae17-d4896fe6c8dc None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:43.223540 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fd4e49-5769-4418-ae17-d4896fe6c8dc None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:43.274880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c1c20b-c814-441a-96d4-c1850cca8bb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:43.275613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c1c20b-c814-441a-96d4-c1850cca8bb3 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:43.275901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c1c20b-c814-441a-96d4-c1850cca8bb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:43.276216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c1c20b-c814-441a-96d4-c1850cca8bb3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:43.300195 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c1c20b-c814-441a-96d4-c1850cca8bb3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:43.304254 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c1c20b-c814-441a-96d4-c1850cca8bb3 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:43.335348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-f3c1c20b-c814-441a-96d4-c1850cca8bb3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:43.335348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:43.335348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:43.335348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:43.335348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:43.335348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:43.335348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:43.335348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:43.335348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:43.335348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:43.335348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:43.335348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:43.335348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:43.335348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:43.335348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:43.335348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:43.338605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:43.338605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:43.338605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:43.338605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:43.338605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:43.338605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:43.338605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:43.338605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:43.338605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:43.338605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:43.338605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:43.338605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:43.338605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:43.338605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:43.338605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:43.338605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:43.338605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:43.340394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:43.340394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:43.340394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:43.340394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 289/562] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:43.349837 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c1c20b-c814-441a-96d4-c1850cca8bb3 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:43.362565 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81247c7-54b9-4a8e-811f-460b87be2bc0 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:43.408303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:43.416505 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35d870e-9802-4e56-966c-7ea51d89702c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:43.417215 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35d870e-9802-4e56-966c-7ea51d89702c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:43.417604 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35d870e-9802-4e56-966c-7ea51d89702c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:43.417760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35d870e-9802-4e56-966c-7ea51d89702c None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:43.422892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35d870e-9802-4e56-966c-7ea51d89702c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:43.422892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35d870e-9802-4e56-966c-7ea51d89702c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:43.422892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35d870e-9802-4e56-966c-7ea51d89702c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:43.456588 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 274/563] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 405 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:43.464506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35d870e-9802-4e56-966c-7ea51d89702c None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:43.526552 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:43.641332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c1c20b-c814-441a-96d4-c1850cca8bb3 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:43.676816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35d870e-9802-4e56-966c-7ea51d89702c None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:43.745590 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cc2b0e-30b8-4c54-a4b1-7033b6414bf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:43.747155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cc2b0e-30b8-4c54-a4b1-7033b6414bf7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:43.749609 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cc2b0e-30b8-4c54-a4b1-7033b6414bf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:43.751088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cc2b0e-30b8-4c54-a4b1-7033b6414bf7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:43.756872 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c139fd-c483-4ebd-b6e7-dd75340994d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:43.758174 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c139fd-c483-4ebd-b6e7-dd75340994d0 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:43.759542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c139fd-c483-4ebd-b6e7-dd75340994d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:43.761198 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c139fd-c483-4ebd-b6e7-dd75340994d0 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:43.762109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c139fd-c483-4ebd-b6e7-dd75340994d0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:43.762678 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c139fd-c483-4ebd-b6e7-dd75340994d0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:43.764869 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c139fd-c483-4ebd-b6e7-dd75340994d0 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:43.769543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90cc2b0e-30b8-4c54-a4b1-7033b6414bf7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:43.772356 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90cc2b0e-30b8-4c54-a4b1-7033b6414bf7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:43.792996 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 290/564] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 449 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:43.799198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-90cc2b0e-30b8-4c54-a4b1-7033b6414bf7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:43.799198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:43.799198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:43.799198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:43.799198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:43.799198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:43.799198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:43.799198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:43.799198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:43.799198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:43.799198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:43.799198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:43.799198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:44:43.799198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:44:43.799198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:44:43.800509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:44:43.800509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:43.800509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:43.800509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:43.800509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:43.800509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:43.800509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:43.800509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:43.800509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:43.800509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:43.800509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:43.800509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:43.800509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:43.800509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:43.800509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:43.800509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:43.800509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:43.801477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:44:43.801477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:44:43.801477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:44:43.801477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:44:43.801477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:44:43.801477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:44:43.801477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:43.801477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:43.803364 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c139fd-c483-4ebd-b6e7-dd75340994d0 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:43.805366 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 275/565] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 341 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:43.828997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cc2b0e-30b8-4c54-a4b1-7033b6414bf7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:43.983527 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:43.992844 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cc2b0e-30b8-4c54-a4b1-7033b6414bf7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:43.995195 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:44.038027 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41b3a3f-fada-45b7-9262-23f38be005e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:44.040286 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41b3a3f-fada-45b7-9262-23f38be005e7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:44.040995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41b3a3f-fada-45b7-9262-23f38be005e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:44.042098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41b3a3f-fada-45b7-9262-23f38be005e7 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:44.043122 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41b3a3f-fada-45b7-9262-23f38be005e7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:44.043875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41b3a3f-fada-45b7-9262-23f38be005e7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:44.046507 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41b3a3f-fada-45b7-9262-23f38be005e7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:44.059854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c139fd-c483-4ebd-b6e7-dd75340994d0 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:44.079335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 276/566] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:44:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:44.088719 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41b3a3f-fada-45b7-9262-23f38be005e7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:44.123428 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643ca654-a23a-47a4-bd16-555a02dd4fb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:44.124160 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643ca654-a23a-47a4-bd16-555a02dd4fb0 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:44.124455 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643ca654-a23a-47a4-bd16-555a02dd4fb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:44.124733 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643ca654-a23a-47a4-bd16-555a02dd4fb0 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:44.133875 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643ca654-a23a-47a4-bd16-555a02dd4fb0 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:44.135582 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643ca654-a23a-47a4-bd16-555a02dd4fb0 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:44.148697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-643ca654-a23a-47a4-bd16-555a02dd4fb0 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 16 00:44:44.148697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:44.148697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:44.148697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:44.148697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:44.148697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:44.148697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:44.148697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:44.148697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:44.148697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:44.148697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:44.148697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:44.148697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:44:44.148697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:44:44.148697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:44:44.148697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:44.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:44.151006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:44.151006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:44.151006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:44:44.151006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:44:44.151006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:44:44.151006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:44:44.151006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:44:44.151006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:44:44.151006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 16 00:44:44.151006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:44.151006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 291/567] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Oct 16 00:44:43 2022] GET /identity/v3/users/nova => generated 82 bytes in 347 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:44.158221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643ca654-a23a-47a4-bd16-555a02dd4fb0 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:44.320106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41b3a3f-fada-45b7-9262-23f38be005e7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:44.374804 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44188590-4364-4ef9-a165-c7b806ede95d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:44.375953 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44188590-4364-4ef9-a165-c7b806ede95d None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:44.376451 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44188590-4364-4ef9-a165-c7b806ede95d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:44.376941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44188590-4364-4ef9-a165-c7b806ede95d None admin] PATH_INFO: `/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/33b136afbf8a4d98a9d62053b68707e9/roles/bfe57c41a4da4b25b9b3e4761f220a46` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:44.395132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44188590-4364-4ef9-a165-c7b806ede95d None admin] RBAC: Authorizing `identity:create_grant(project_id=7deb2c3dae214df7895b40e7339bd6ea, user_id=33b136afbf8a4d98a9d62053b68707e9, role_id=bfe57c41a4da4b25b9b3e4761f220a46)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:44.396997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44188590-4364-4ef9-a165-c7b806ede95d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:44.402690 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643ca654-a23a-47a4-bd16-555a02dd4fb0 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:44.423146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 277/568] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:44:44 2022] PUT /identity/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/33b136afbf8a4d98a9d62053b68707e9/roles/bfe57c41a4da4b25b9b3e4761f220a46 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:44:44.457289 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af36af1-07c4-44c0-a187-0290358d0802 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:44.458326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af36af1-07c4-44c0-a187-0290358d0802 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:44.458819 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af36af1-07c4-44c0-a187-0290358d0802 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:44.459339 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af36af1-07c4-44c0-a187-0290358d0802 None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:44.460327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af36af1-07c4-44c0-a187-0290358d0802 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:44.460865 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af36af1-07c4-44c0-a187-0290358d0802 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:44.462794 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af36af1-07c4-44c0-a187-0290358d0802 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:44.468048 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:44.489981 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 292/569] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Oct 16 00:44:44 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 332 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:44.499472 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44188590-4364-4ef9-a165-c7b806ede95d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:44.525480 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:44.699649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44188590-4364-4ef9-a165-c7b806ede95d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:44.740292 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a10f6bf-168b-4380-8432-341adb4f7cf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:44.740936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a10f6bf-168b-4380-8432-341adb4f7cf2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:44.741218 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a10f6bf-168b-4380-8432-341adb4f7cf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:44.741487 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a10f6bf-168b-4380-8432-341adb4f7cf2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:44.820558 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a10f6bf-168b-4380-8432-341adb4f7cf2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:44.822048 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a10f6bf-168b-4380-8432-341adb4f7cf2 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:44.831828 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:44.932887 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 278/570] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:44:44 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 16 00:44:44.941943 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af36af1-07c4-44c0-a187-0290358d0802 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:45.177728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af36af1-07c4-44c0-a187-0290358d0802 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:45.241940 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0438bf48-39c6-4a46-a974-bd59c439e335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:45.243256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0438bf48-39c6-4a46-a974-bd59c439e335 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:45.243744 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0438bf48-39c6-4a46-a974-bd59c439e335 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:45.244233 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0438bf48-39c6-4a46-a974-bd59c439e335 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:45.264083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0438bf48-39c6-4a46-a974-bd59c439e335 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:45.266805 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0438bf48-39c6-4a46-a974-bd59c439e335 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:45.295703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-0438bf48-39c6-4a46-a974-bd59c439e335 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:45.295703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:45.295703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:45.295703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:45.295703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:45.295703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:45.295703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:45.295703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:45.295703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:45.295703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:45.295703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:45.295703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:45.295703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:45.295703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:45.295703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:45.295703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:45.296823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:45.296823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:45.296823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:45.296823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:45.296823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:45.296823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:45.296823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:45.296823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:45.296823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:45.296823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:45.296823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:45.296823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:45.296823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:45.296823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:45.296823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:45.296823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:45.296823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:45.297725 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:45.297725 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:45.297725 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:45.298429 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 293/571] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 357 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:45.309209 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a10f6bf-168b-4380-8432-341adb4f7cf2 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:45.340715 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4f29f9-cd29-4330-8cae-45f5d0fdd205 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:45.341185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4f29f9-cd29-4330-8cae-45f5d0fdd205 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:45.341637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4f29f9-cd29-4330-8cae-45f5d0fdd205 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:45.346888 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 294/572] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:45 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:44:45.354351 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a54bde5-65c6-414f-99c7-1e8e0bb8c989 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:45.354774 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a54bde5-65c6-414f-99c7-1e8e0bb8c989 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:45.359636 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a54bde5-65c6-414f-99c7-1e8e0bb8c989 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:45.398212 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-9a54bde5-65c6-414f-99c7-1e8e0bb8c989 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:45.400855 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-9a54bde5-65c6-414f-99c7-1e8e0bb8c989 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:45.504879 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:45.571610 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 295/573] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:45 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:44:45.621357 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a10f6bf-168b-4380-8432-341adb4f7cf2 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:45.638617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:45.712445 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb64f6e2-c1a2-4625-b11e-ee0093558c92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:45.713514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb64f6e2-c1a2-4625-b11e-ee0093558c92 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:45.713979 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb64f6e2-c1a2-4625-b11e-ee0093558c92 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:45.714460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb64f6e2-c1a2-4625-b11e-ee0093558c92 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:45.715628 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb64f6e2-c1a2-4625-b11e-ee0093558c92 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:45.716176 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb64f6e2-c1a2-4625-b11e-ee0093558c92 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:45.718696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb64f6e2-c1a2-4625-b11e-ee0093558c92 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:45.746264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 279/574] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 440 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:45.755478 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a54bde5-65c6-414f-99c7-1e8e0bb8c989 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:45.786887 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1090afa-a5d1-435f-9a20-38d68fe6dbca None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:45.787348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1090afa-a5d1-435f-9a20-38d68fe6dbca None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:45.787798 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1090afa-a5d1-435f-9a20-38d68fe6dbca None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:45.790767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 280/575] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:45 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:44:45.797651 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f327296d-be6d-4741-8b1e-0d021fec8eba None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:45.798107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f327296d-be6d-4741-8b1e-0d021fec8eba None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:45.798565 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f327296d-be6d-4741-8b1e-0d021fec8eba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:45.848723 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f327296d-be6d-4741-8b1e-0d021fec8eba None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:45.850662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f327296d-be6d-4741-8b1e-0d021fec8eba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:45.966123 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a54bde5-65c6-414f-99c7-1e8e0bb8c989 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:46.036144 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712f38a1-ba76-47e5-9352-ff7c58aaa26c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:46.037218 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712f38a1-ba76-47e5-9352-ff7c58aaa26c None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:46.037716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712f38a1-ba76-47e5-9352-ff7c58aaa26c None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:46.038202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712f38a1-ba76-47e5-9352-ff7c58aaa26c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:46.046901 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 281/576] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:45 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:44:46.048076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f327296d-be6d-4741-8b1e-0d021fec8eba None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:46.052750 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712f38a1-ba76-47e5-9352-ff7c58aaa26c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:46.055544 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712f38a1-ba76-47e5-9352-ff7c58aaa26c None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:46.080300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-712f38a1-ba76-47e5-9352-ff7c58aaa26c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:46.080300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:46.080300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:46.080300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:46.080300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:46.080300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:46.080300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:46.080300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:46.080300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:46.080300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:46.080300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:46.080300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:46.080300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:44:46.080300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:44:46.080300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:44:46.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:44:46.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:46.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:46.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:46.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:46.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:46.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:46.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:46.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:46.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:46.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:46.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:46.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:46.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:46.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:46.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:46.081337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:46.082294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:44:46.082294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:44:46.082294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:44:46.082294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:44:46.082294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:44:46.082294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:44:46.082294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:46.082294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:46.082771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 296/577] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 328 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:46.093267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712f38a1-ba76-47e5-9352-ff7c58aaa26c None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:46.214062 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:46.338030 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:46.351358 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712f38a1-ba76-47e5-9352-ff7c58aaa26c None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:46.355140 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f327296d-be6d-4741-8b1e-0d021fec8eba None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:46.396174 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897f5388-ca3c-44c5-90ea-9144ffa3b5cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:46.396867 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897f5388-ca3c-44c5-90ea-9144ffa3b5cd None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:46.397150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897f5388-ca3c-44c5-90ea-9144ffa3b5cd None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:46.397433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897f5388-ca3c-44c5-90ea-9144ffa3b5cd None admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:46.398164 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897f5388-ca3c-44c5-90ea-9144ffa3b5cd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:46.399667 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897f5388-ca3c-44c5-90ea-9144ffa3b5cd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:46.401460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897f5388-ca3c-44c5-90ea-9144ffa3b5cd None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:46.423879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b92f3a-1565-4117-97b6-0df081051c69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:46.424203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 297/578] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:44:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 332 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:46.424895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b92f3a-1565-4117-97b6-0df081051c69 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:46.425010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897f5388-ca3c-44c5-90ea-9144ffa3b5cd None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:46.425330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b92f3a-1565-4117-97b6-0df081051c69 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:46.425870 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b92f3a-1565-4117-97b6-0df081051c69 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:46.452976 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b92f3a-1565-4117-97b6-0df081051c69 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:46.467614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-c2b92f3a-1565-4117-97b6-0df081051c69 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:44:46.467614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:46.467614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:46.467614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:46.467614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:46.467614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:46.467614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:46.467614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:46.467614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:46.467614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:46.467614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:46.467614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:46.467614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:46.467614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:46.467614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:46.467614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:46.468727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:46.468727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:46.468727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:46.468727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:46.468727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:46.468727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:46.468727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:46.468727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:46.468727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:46.468727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:46.468727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:46.468727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:46.468727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:46.468727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:46.468727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:46.468727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:46.469707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:46.469707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:46.469707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:46.469707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:46.469707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:46.469707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:46.469707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:46.469707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:46.469707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:46.469707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:46.469707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:44:46.469707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:46.475338 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 282/579] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:44:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 424 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:46.475338 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b92f3a-1565-4117-97b6-0df081051c69 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:46.599623 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:46.613266 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897f5388-ca3c-44c5-90ea-9144ffa3b5cd None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:46.656925 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d896323-32a0-444a-ac71-0dd57e3883a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:46.657574 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d896323-32a0-444a-ac71-0dd57e3883a0 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:46.657888 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d896323-32a0-444a-ac71-0dd57e3883a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:46.658161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d896323-32a0-444a-ac71-0dd57e3883a0 None admin] PATH_INFO: `/v3/services/network` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:46.658348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b92f3a-1565-4117-97b6-0df081051c69 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:46.658720 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d896323-32a0-444a-ac71-0dd57e3883a0 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:46.660149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d896323-32a0-444a-ac71-0dd57e3883a0 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:46.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-7d896323-32a0-444a-ac71-0dd57e3883a0 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 16 00:44:46.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:46.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:44:46.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:44:46.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:44:46.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:44:46.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:44:46.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:46.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 16 00:44:46.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:46.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:44:46.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:46.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:46.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:46.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:46.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:46.678388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:46.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:46.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:46.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:46.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:46.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:46.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:46.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:44:46.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:44:46.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:44:46.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:44:46.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:46.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:46.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:46.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:46.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:46.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:46.680734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:46.680734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:46.680734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:46.680734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:46.680734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:46.680734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:46.680734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:46.680734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:46.680734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:46.680734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:46.680734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:44:46.680734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:46.680734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 16 00:44:46.680734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:46.686817 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 298/580] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:46 2022] GET /identity/v3/services/network => generated 88 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:46.686817 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d896323-32a0-444a-ac71-0dd57e3883a0 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:46.708349 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79babaf5-a4a0-47c7-bf7b-b98205144c6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:46.709137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79babaf5-a4a0-47c7-bf7b-b98205144c6e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:46.709635 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79babaf5-a4a0-47c7-bf7b-b98205144c6e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:46.710124 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79babaf5-a4a0-47c7-bf7b-b98205144c6e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:46.713178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79babaf5-a4a0-47c7-bf7b-b98205144c6e None admin] RBAC: Adding query filter params (role.id=c5cd777d70a649028f970fd986dc484f, scope.project.id=7deb2c3dae214df7895b40e7339bd6ea, user.id=4eba26813e154595a12a67fe4be5eeff) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:46.713701 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79babaf5-a4a0-47c7-bf7b-b98205144c6e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:46.716544 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79babaf5-a4a0-47c7-bf7b-b98205144c6e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:46.728824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 283/581] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:44:46 2022] GET /identity/v3/role_assignments?user.id=4eba26813e154595a12a67fe4be5eeff&scope.project.id=7deb2c3dae214df7895b40e7339bd6ea&role.id=c5cd777d70a649028f970fd986dc484f => generated 608 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:46.730971 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61e9d1d-ce78-4448-b66d-1a4ab1ecbb9a None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:46.731397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61e9d1d-ce78-4448-b66d-1a4ab1ecbb9a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:46.731881 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61e9d1d-ce78-4448-b66d-1a4ab1ecbb9a None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:46.736403 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 284/582] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:46 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:46.736900 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61e9d1d-ce78-4448-b66d-1a4ab1ecbb9a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:46.792832 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:46.897050 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d896323-32a0-444a-ac71-0dd57e3883a0 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:46.973209 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58456fc7-5302-4de3-91ad-9f2424d551a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:46.974953 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58456fc7-5302-4de3-91ad-9f2424d551a6 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:46.975814 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58456fc7-5302-4de3-91ad-9f2424d551a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:46.977333 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58456fc7-5302-4de3-91ad-9f2424d551a6 None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:46.978981 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58456fc7-5302-4de3-91ad-9f2424d551a6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:46.979832 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58456fc7-5302-4de3-91ad-9f2424d551a6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:46.982503 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58456fc7-5302-4de3-91ad-9f2424d551a6 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:46.998512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 299/583] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:44:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:46.999695 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5859c3ad-b869-482a-86ed-221afe3a4d7e None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:46.999985 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5859c3ad-b869-482a-86ed-221afe3a4d7e None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:47.000261 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5859c3ad-b869-482a-86ed-221afe3a4d7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:47.012893 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61e9d1d-ce78-4448-b66d-1a4ab1ecbb9a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:47.029709 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-5859c3ad-b869-482a-86ed-221afe3a4d7e None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:47.032233 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-5859c3ad-b869-482a-86ed-221afe3a4d7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:47.062518 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:47.086769 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252b4b9d-3297-4c10-af5b-0d6c1eb967c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:47.088366 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252b4b9d-3297-4c10-af5b-0d6c1eb967c4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:47.089336 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252b4b9d-3297-4c10-af5b-0d6c1eb967c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:47.090084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252b4b9d-3297-4c10-af5b-0d6c1eb967c4 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:47.091689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-252b4b9d-3297-4c10-af5b-0d6c1eb967c4 None admin] RBAC: Adding query filter params (name=network) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:47.092575 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-252b4b9d-3297-4c10-af5b-0d6c1eb967c4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:47.095960 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-252b4b9d-3297-4c10-af5b-0d6c1eb967c4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:47.111049 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 285/584] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 16 00:44:46 2022] GET /identity/v3/services?name=network => generated 124 bytes in 370 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:47.112698 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252b4b9d-3297-4c10-af5b-0d6c1eb967c4 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:47.179663 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 300/585] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:46 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:44:47.179663 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5859c3ad-b869-482a-86ed-221afe3a4d7e None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:47.314297 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:47.342115 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252b4b9d-3297-4c10-af5b-0d6c1eb967c4 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:47.371616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5859c3ad-b869-482a-86ed-221afe3a4d7e None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:47.387540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6913f0-acae-426f-9356-7741c111b557 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:47.388569 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6913f0-acae-426f-9356-7741c111b557 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:47.389057 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6913f0-acae-426f-9356-7741c111b557 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:47.389554 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6913f0-acae-426f-9356-7741c111b557 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:47.401705 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6913f0-acae-426f-9356-7741c111b557 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:47.403741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6913f0-acae-426f-9356-7741c111b557 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:47.421047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-dd6913f0-acae-426f-9356-7741c111b557 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:47.421047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:47.421047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:47.421047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:47.421047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:47.421047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:47.421047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:47.421047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:47.421047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:47.421047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:47.421047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:47.421047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:47.421047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:47.421047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:47.421047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:47.421047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:47.422158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:47.422158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:47.422158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:47.422158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:47.422158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:47.422158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:47.422158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:47.422158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:47.422158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:47.422158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:47.422158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:47.422158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:47.422158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:47.422158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:47.422158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:47.422158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:47.422158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:47.423115 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:47.423115 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:47.423115 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:47.425592 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 286/586] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:47.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df88b55b-bb4f-4d12-8abb-95c9d8307679 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:47.427829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df88b55b-bb4f-4d12-8abb-95c9d8307679 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:47.428297 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df88b55b-bb4f-4d12-8abb-95c9d8307679 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:47.430464 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 287/587] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:47.436071 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac9f754-d0a7-4343-a8ea-1088ae7da216 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:47.436071 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac9f754-d0a7-4343-a8ea-1088ae7da216 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:47.438397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac9f754-d0a7-4343-a8ea-1088ae7da216 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:47.442041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4effde07-7e07-4cbd-8f7c-1f1ccb6d4b7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:47.443110 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4effde07-7e07-4cbd-8f7c-1f1ccb6d4b7c None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:47.443705 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4effde07-7e07-4cbd-8f7c-1f1ccb6d4b7c None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:47.444774 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4effde07-7e07-4cbd-8f7c-1f1ccb6d4b7c None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:47.446350 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4effde07-7e07-4cbd-8f7c-1f1ccb6d4b7c None admin] RBAC: Adding query filter params (type=network) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:47.446765 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4effde07-7e07-4cbd-8f7c-1f1ccb6d4b7c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:47.449115 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4effde07-7e07-4cbd-8f7c-1f1ccb6d4b7c None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:47.456528 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5ac9f754-d0a7-4343-a8ea-1088ae7da216 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:47.457750 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5ac9f754-d0a7-4343-a8ea-1088ae7da216 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:47.458694 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 301/588] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 16 00:44:47 2022] GET /identity/v3/services?type=network => generated 350 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:47.460400 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4effde07-7e07-4cbd-8f7c-1f1ccb6d4b7c None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:47.490605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:47.579504 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 288/589] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:47 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:44:47.581403 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac9f754-d0a7-4343-a8ea-1088ae7da216 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:47.691180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4effde07-7e07-4cbd-8f7c-1f1ccb6d4b7c None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:47.724915 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:47.777111 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f3b6c6-a1bd-4b9d-b2ca-ffd1d692831e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:47.778117 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f3b6c6-a1bd-4b9d-b2ca-ffd1d692831e None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:47.778621 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f3b6c6-a1bd-4b9d-b2ca-ffd1d692831e None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:47.779152 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f3b6c6-a1bd-4b9d-b2ca-ffd1d692831e None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:47.781144 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f3b6c6-a1bd-4b9d-b2ca-ffd1d692831e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:47.781703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f3b6c6-a1bd-4b9d-b2ca-ffd1d692831e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:47.784901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f3b6c6-a1bd-4b9d-b2ca-ffd1d692831e None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:47.804345 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac9f754-d0a7-4343-a8ea-1088ae7da216 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:47.815363 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 302/590] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 356 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:47.817013 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f3b6c6-a1bd-4b9d-b2ca-ffd1d692831e None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:47.886559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c1a4c6-4378-4aa5-99e2-32f2902bc5b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:47.888182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c1a4c6-4378-4aa5-99e2-32f2902bc5b7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:47.888987 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c1a4c6-4378-4aa5-99e2-32f2902bc5b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:47.889787 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c1a4c6-4378-4aa5-99e2-32f2902bc5b7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:47.891688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c1a4c6-4378-4aa5-99e2-32f2902bc5b7 None admin] RBAC: Adding query filter params (interface=public, service_id=236eb3386fe84842a899315e5040ec1b, region_id=RegionOne) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:47.892274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c1a4c6-4378-4aa5-99e2-32f2902bc5b7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:47.895004 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c1a4c6-4378-4aa5-99e2-32f2902bc5b7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:47.910355 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 289/591] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Oct 16 00:44:47 2022] GET /identity/v3/endpoints?service_id=236eb3386fe84842a899315e5040ec1b&interface=public®ion_id=RegionOne => generated 194 bytes in 330 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:47.911575 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c1a4c6-4378-4aa5-99e2-32f2902bc5b7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:47.969968 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:48.018698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f3b6c6-a1bd-4b9d-b2ca-ffd1d692831e None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:48.067442 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8471dfb-c78d-4eb1-8eca-a7fde023d8bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:48.068701 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8471dfb-c78d-4eb1-8eca-a7fde023d8bd None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:48.069420 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8471dfb-c78d-4eb1-8eca-a7fde023d8bd None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:48.069929 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8471dfb-c78d-4eb1-8eca-a7fde023d8bd None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:48.072091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8471dfb-c78d-4eb1-8eca-a7fde023d8bd None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:48.075315 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8471dfb-c78d-4eb1-8eca-a7fde023d8bd None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:48.076715 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c1a4c6-4378-4aa5-99e2-32f2902bc5b7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:48.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-d8471dfb-c78d-4eb1-8eca-a7fde023d8bd None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 16 00:44:48.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:48.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:44:48.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:44:48.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:44:48.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:44:48.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:44:48.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:48.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 16 00:44:48.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:48.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:44:48.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:48.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:48.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:48.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:48.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:48.094485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:48.096783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:48.096783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:48.096783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:48.096783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:48.096783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:48.096783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:48.096783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:44:48.096783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:44:48.096783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:44:48.096783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:44:48.096783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:48.096783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:48.096783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:48.096783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:48.096783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:48.096783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:48.098706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:48.098706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:48.098706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:48.098706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:48.098706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:48.098706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:48.098706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:48.098706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:48.098706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:48.098706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:48.098706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:44:48.098706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:48.098706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 16 00:44:48.098706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:48.103631 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 303/592] 10.222.0.25 () {56 vars in 1129 bytes} [Sun Oct 16 00:44:47 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:48.105420 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8471dfb-c78d-4eb1-8eca-a7fde023d8bd None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:48.109805 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:48.120105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d8ea27-8171-4b9d-92b1-de7bd702dee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:48.121110 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d8ea27-8171-4b9d-92b1-de7bd702dee9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:48.121583 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d8ea27-8171-4b9d-92b1-de7bd702dee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:48.122048 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d8ea27-8171-4b9d-92b1-de7bd702dee9 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:48.135694 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d8ea27-8171-4b9d-92b1-de7bd702dee9 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:48.137987 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d8ea27-8171-4b9d-92b1-de7bd702dee9 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:48.157581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-54d8ea27-8171-4b9d-92b1-de7bd702dee9 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 16 00:44:48.157581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:48.157581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:48.157581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:48.157581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:48.157581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:48.157581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:48.157581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:48.157581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:48.157581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:48.157581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:48.157581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:48.157581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:44:48.157581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:44:48.157581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:44:48.157581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:48.159760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:48.161614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:48.161614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:48.161614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:44:48.161614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:44:48.161614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:44:48.161614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:44:48.161614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:44:48.161614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:44:48.161614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 16 00:44:48.161614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:48.161614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 290/593] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:44:47 2022] GET /identity/v3/users/swift => generated 83 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:48.161614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d8ea27-8171-4b9d-92b1-de7bd702dee9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:48.309465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8471dfb-c78d-4eb1-8eca-a7fde023d8bd None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:48.320680 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d8ea27-8171-4b9d-92b1-de7bd702dee9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:48.369215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf43deb-a9b3-4bc7-86e8-ad98cef95d34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:48.369867 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf43deb-a9b3-4bc7-86e8-ad98cef95d34 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:48.370151 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf43deb-a9b3-4bc7-86e8-ad98cef95d34 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:48.370458 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf43deb-a9b3-4bc7-86e8-ad98cef95d34 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:48.372450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf9543a-a8ae-4549-b235-43e4f46a9612 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:48.373498 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf9543a-a8ae-4549-b235-43e4f46a9612 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:48.373987 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf9543a-a8ae-4549-b235-43e4f46a9612 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:48.374477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf9543a-a8ae-4549-b235-43e4f46a9612 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:48.375624 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf9543a-a8ae-4549-b235-43e4f46a9612 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:48.378237 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf9543a-a8ae-4549-b235-43e4f46a9612 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:48.379518 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:48.385596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf43deb-a9b3-4bc7-86e8-ad98cef95d34 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:48.391276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-fbf43deb-a9b3-4bc7-86e8-ad98cef95d34 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:48.391276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:48.391276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:48.391276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:48.391276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:48.391276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:48.391276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:48.391276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:48.391276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:48.391276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:48.391276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:48.391276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:48.391276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:48.391276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:48.391276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:48.391276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:48.393405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:48.393405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:48.393405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:48.393405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:48.393405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:48.393405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:48.393405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:48.393405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:48.393405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:48.393405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:48.393405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:48.393405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:48.393405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:48.393405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:48.393405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:48.393405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:48.395633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:48.395633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:48.395633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:48.395633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:48.395633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:48.395633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:48.395633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:48.395633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:48.395633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:48.395633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:48.395633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:44:48.395633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:48.403568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 291/594] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 16 00:44:48 2022] GET /identity/v3/services => generated 772 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:48.404413 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 304/595] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 16 00:44:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:48.405527 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf43deb-a9b3-4bc7-86e8-ad98cef95d34 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:48.405691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf9543a-a8ae-4549-b235-43e4f46a9612 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:48.508967 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:48.572806 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf43deb-a9b3-4bc7-86e8-ad98cef95d34 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:48.601223 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf9543a-a8ae-4549-b235-43e4f46a9612 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:48.626758 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26287fe-be53-454a-aba4-9c2f11c88fd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:48.627832 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26287fe-be53-454a-aba4-9c2f11c88fd5 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:48.628613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26287fe-be53-454a-aba4-9c2f11c88fd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:48.629115 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26287fe-be53-454a-aba4-9c2f11c88fd5 None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:48.630803 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26287fe-be53-454a-aba4-9c2f11c88fd5 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:48.631407 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26287fe-be53-454a-aba4-9c2f11c88fd5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:48.633624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26287fe-be53-454a-aba4-9c2f11c88fd5 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:48.659385 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 305/596] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Oct 16 00:44:48 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:48.660935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26287fe-be53-454a-aba4-9c2f11c88fd5 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:48.674093 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a818dc-e563-40ba-ac72-34ecb44565bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:48.675292 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a818dc-e563-40ba-ac72-34ecb44565bc None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:48.675991 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a818dc-e563-40ba-ac72-34ecb44565bc None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:48.676294 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a818dc-e563-40ba-ac72-34ecb44565bc None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:48.677429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a818dc-e563-40ba-ac72-34ecb44565bc None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:48.677992 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a818dc-e563-40ba-ac72-34ecb44565bc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:48.681277 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a818dc-e563-40ba-ac72-34ecb44565bc None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:48.699016 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 292/597] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Oct 16 00:44:48 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:48.700808 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a818dc-e563-40ba-ac72-34ecb44565bc None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:48.747247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:48.858094 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26287fe-be53-454a-aba4-9c2f11c88fd5 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:48.932063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6651fbb-a7d8-4152-874b-f8afa7880e55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:48.933025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a818dc-e563-40ba-ac72-34ecb44565bc None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:48.933829 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6651fbb-a7d8-4152-874b-f8afa7880e55 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:48.934620 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6651fbb-a7d8-4152-874b-f8afa7880e55 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:48.935408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6651fbb-a7d8-4152-874b-f8afa7880e55 None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:48.938782 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6651fbb-a7d8-4152-874b-f8afa7880e55 None admin] RBAC: Adding query filter params (name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:48.939764 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6651fbb-a7d8-4152-874b-f8afa7880e55 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:48.942773 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6651fbb-a7d8-4152-874b-f8afa7880e55 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:48.971156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 306/598] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 16 00:44:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:48.974931 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6651fbb-a7d8-4152-874b-f8afa7880e55 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:48.975799 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:49.012020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3335cd-1a3b-43db-97a0-43a6db6827a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:49.013144 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3335cd-1a3b-43db-97a0-43a6db6827a4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:49.013631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3335cd-1a3b-43db-97a0-43a6db6827a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:49.014549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3335cd-1a3b-43db-97a0-43a6db6827a4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:49.157615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3335cd-1a3b-43db-97a0-43a6db6827a4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:49.161740 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3335cd-1a3b-43db-97a0-43a6db6827a4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:49.223959 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6651fbb-a7d8-4152-874b-f8afa7880e55 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:49.262398 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:49.299344 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afeb855-77e8-45ba-88d1-f2f556e0a2d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:49.300439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afeb855-77e8-45ba-88d1-f2f556e0a2d9 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:49.301020 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afeb855-77e8-45ba-88d1-f2f556e0a2d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:49.301505 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afeb855-77e8-45ba-88d1-f2f556e0a2d9 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:49.302848 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afeb855-77e8-45ba-88d1-f2f556e0a2d9 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:49.303687 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afeb855-77e8-45ba-88d1-f2f556e0a2d9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:49.307439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afeb855-77e8-45ba-88d1-f2f556e0a2d9 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:49.317223 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 307/599] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Oct 16 00:44:48 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 345 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:49.325113 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afeb855-77e8-45ba-88d1-f2f556e0a2d9 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:49.406455 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 293/600] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:44:48 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 707 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 16 00:44:49.408383 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3335cd-1a3b-43db-97a0-43a6db6827a4 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:49.452749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:49.583305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afeb855-77e8-45ba-88d1-f2f556e0a2d9 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:49.589408 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3335cd-1a3b-43db-97a0-43a6db6827a4 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:49.643520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c45e3f1-81e4-44aa-9e62-03061165e33f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:49.644331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c45e3f1-81e4-44aa-9e62-03061165e33f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:49.644773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c45e3f1-81e4-44aa-9e62-03061165e33f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:49.645201 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c45e3f1-81e4-44aa-9e62-03061165e33f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:49.646049 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c45e3f1-81e4-44aa-9e62-03061165e33f None admin] RBAC: Adding query filter params (interface=public, service_id=3a057bb5a2cb420193e5086ef92846a0, region_id=RegionOne) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:49.646212 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e214c1a4-fc69-4dcb-b3f5-07e459319d91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:49.646563 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c45e3f1-81e4-44aa-9e62-03061165e33f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:49.647241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e214c1a4-fc69-4dcb-b3f5-07e459319d91 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:49.647751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e214c1a4-fc69-4dcb-b3f5-07e459319d91 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:49.648251 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e214c1a4-fc69-4dcb-b3f5-07e459319d91 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:49.648571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c45e3f1-81e4-44aa-9e62-03061165e33f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:49.661731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 294/601] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Oct 16 00:44:49 2022] GET /identity/v3/endpoints?service_id=3a057bb5a2cb420193e5086ef92846a0&interface=public®ion_id=RegionOne => generated 194 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:49.661731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c45e3f1-81e4-44aa-9e62-03061165e33f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:49.671604 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e214c1a4-fc69-4dcb-b3f5-07e459319d91 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:49.674598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e214c1a4-fc69-4dcb-b3f5-07e459319d91 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:49.703124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-e214c1a4-fc69-4dcb-b3f5-07e459319d91 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:49.703124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:49.703124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:49.703124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:49.703124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:49.703124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:49.703124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:49.703124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:49.703124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:49.703124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:49.703124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:49.703124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:49.703124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:49.703124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:49.703124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:49.703124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:49.705311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:49.705311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:49.705311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:49.705311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:49.705311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:49.705311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:49.705311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:49.705311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:49.705311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:49.705311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:49.705311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:49.705311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:49.705311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:49.705311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:49.705311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:49.705311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:49.705311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:49.707118 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:49.707118 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:49.707118 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:49.707118 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 308/602] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 382 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:49.708371 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f8acac-f642-401f-8a44-ef583cea47a3 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:49.708371 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f8acac-f642-401f-8a44-ef583cea47a3 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:49.708828 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f8acac-f642-401f-8a44-ef583cea47a3 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:49.711436 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 309/603] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:49 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:49.712852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f8acac-f642-401f-8a44-ef583cea47a3 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:49.781835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:49.785056 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:49.821454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c45e3f1-81e4-44aa-9e62-03061165e33f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:49.888937 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576b1188-d453-4119-82ff-0e42bcf7109f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:49.890065 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576b1188-d453-4119-82ff-0e42bcf7109f None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:49.890569 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576b1188-d453-4119-82ff-0e42bcf7109f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:49.891131 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576b1188-d453-4119-82ff-0e42bcf7109f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:49.906593 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f8acac-f642-401f-8a44-ef583cea47a3 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:49.914918 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576b1188-d453-4119-82ff-0e42bcf7109f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:49.917879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576b1188-d453-4119-82ff-0e42bcf7109f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:49.950489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-576b1188-d453-4119-82ff-0e42bcf7109f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:49.950489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:49.950489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:49.950489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:49.950489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:49.950489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:49.950489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:49.950489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:49.950489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:49.950489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:49.950489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:49.950489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:49.950489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:49.950489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:49.950489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:49.950489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:49.952422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:49.952422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:49.952422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:49.952422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:49.952422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:49.952422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:49.952422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:49.952422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:49.952422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:49.952422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:49.952422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:49.952422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:49.952422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:49.952422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:49.952422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:49.952422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:49.952422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:49.961995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:49.961995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:49.961995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:49.962359 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdcd6b7-17ec-457d-bcd0-a33288510343 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:49.962359 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdcd6b7-17ec-457d-bcd0-a33288510343 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:49.962359 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdcd6b7-17ec-457d-bcd0-a33288510343 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:49.962359 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdcd6b7-17ec-457d-bcd0-a33288510343 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:49.962962 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdcd6b7-17ec-457d-bcd0-a33288510343 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:49.963109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 295/604] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:49.963275 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdcd6b7-17ec-457d-bcd0-a33288510343 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:49.963424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576b1188-d453-4119-82ff-0e42bcf7109f None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:49.970460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 310/605] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 16 00:44:49 2022] GET /identity/v3/services => generated 772 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:49.978944 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ffebcb-e147-493c-b2b1-d17258ecd82c None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:49.978944 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ffebcb-e147-493c-b2b1-d17258ecd82c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:49.978944 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ffebcb-e147-493c-b2b1-d17258ecd82c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:50.003588 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-81ffebcb-e147-493c-b2b1-d17258ecd82c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:50.004999 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-81ffebcb-e147-493c-b2b1-d17258ecd82c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:50.120066 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 311/606] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:49 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:44:50.121791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ffebcb-e147-493c-b2b1-d17258ecd82c None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:50.147850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576b1188-d453-4119-82ff-0e42bcf7109f None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:50.177990 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:50.195860 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c724dc-7eae-4b78-8960-37ee3f896deb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:50.196527 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c724dc-7eae-4b78-8960-37ee3f896deb None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:50.196986 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c724dc-7eae-4b78-8960-37ee3f896deb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:50.197267 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c724dc-7eae-4b78-8960-37ee3f896deb None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:50.198005 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c724dc-7eae-4b78-8960-37ee3f896deb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:50.198318 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c724dc-7eae-4b78-8960-37ee3f896deb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:50.199920 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c724dc-7eae-4b78-8960-37ee3f896deb None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:50.201636 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:50.217797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 296/607] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:50.223552 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c724dc-7eae-4b78-8960-37ee3f896deb None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:50.333078 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ffebcb-e147-493c-b2b1-d17258ecd82c None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:50.375607 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c724dc-7eae-4b78-8960-37ee3f896deb None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:50.410902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292ae86c-4122-425f-a85e-3c1f6f0c7fc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:50.410902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292ae86c-4122-425f-a85e-3c1f6f0c7fc5 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:50.410902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292ae86c-4122-425f-a85e-3c1f6f0c7fc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:50.410902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292ae86c-4122-425f-a85e-3c1f6f0c7fc5 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:50.411861 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292ae86c-4122-425f-a85e-3c1f6f0c7fc5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:50.413921 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292ae86c-4122-425f-a85e-3c1f6f0c7fc5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:50.416048 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292ae86c-4122-425f-a85e-3c1f6f0c7fc5 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:50.419382 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5daf3033-e3d5-4965-85d5-f1cfd5fea891 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:50.420055 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5daf3033-e3d5-4965-85d5-f1cfd5fea891 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:50.420325 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5daf3033-e3d5-4965-85d5-f1cfd5fea891 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:50.420701 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5daf3033-e3d5-4965-85d5-f1cfd5fea891 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:50.430899 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5daf3033-e3d5-4965-85d5-f1cfd5fea891 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:50.432136 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5daf3033-e3d5-4965-85d5-f1cfd5fea891 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:50.445116 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 312/608] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:50.447758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-5daf3033-e3d5-4965-85d5-f1cfd5fea891 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 16 00:44:50.447758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:50.447758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:50.447758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:50.447758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:50.447758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:50.447758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:50.447758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:50.447758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:50.447758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:50.447758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:50.447758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:50.447758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:44:50.447758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:44:50.447758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:44:50.447758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:50.448725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:50.449702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:50.449702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:50.449702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:44:50.449702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:44:50.449702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:44:50.449702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:44:50.449702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:44:50.449702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:44:50.449702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 16 00:44:50.449702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:50.452430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 297/609] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 16 00:44:50 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:50.457267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292ae86c-4122-425f-a85e-3c1f6f0c7fc5 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:50.459142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5daf3033-e3d5-4965-85d5-f1cfd5fea891 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:50.546969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:50.638667 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:50.667205 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292ae86c-4122-425f-a85e-3c1f6f0c7fc5 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:50.684727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5daf3033-e3d5-4965-85d5-f1cfd5fea891 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:50.730294 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88988500-63c7-458c-9505-b6abad755d8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:50.733086 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88988500-63c7-458c-9505-b6abad755d8a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:50.735101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88988500-63c7-458c-9505-b6abad755d8a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:50.735605 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88988500-63c7-458c-9505-b6abad755d8a None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:50.736614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88988500-63c7-458c-9505-b6abad755d8a None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:50.737391 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88988500-63c7-458c-9505-b6abad755d8a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:50.740340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88988500-63c7-458c-9505-b6abad755d8a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:50.772540 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7de1bb-02ad-419d-ae3e-3b5dde862e02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:50.774435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7de1bb-02ad-419d-ae3e-3b5dde862e02 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:50.775011 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7de1bb-02ad-419d-ae3e-3b5dde862e02 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:50.775480 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7de1bb-02ad-419d-ae3e-3b5dde862e02 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:50.781427 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 298/610] 10.222.0.25 () {56 vars in 1149 bytes} [Sun Oct 16 00:44:50 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 323 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:50.787326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88988500-63c7-458c-9505-b6abad755d8a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:50.792463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7de1bb-02ad-419d-ae3e-3b5dde862e02 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:50.794738 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7de1bb-02ad-419d-ae3e-3b5dde862e02 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:50.812250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-3e7de1bb-02ad-419d-ae3e-3b5dde862e02 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:50.812250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:50.812250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:50.812250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:50.812250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:50.812250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:50.812250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:50.812250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:50.812250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:50.812250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:50.812250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:50.812250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:50.812250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:44:50.812250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:44:50.812250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:44:50.813252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:44:50.813252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:50.813252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:50.813252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:50.813252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:50.813252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:50.813252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:50.813252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:50.813252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:50.813252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:50.813252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:50.813252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:50.813252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:50.813252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:50.813252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:50.813252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:50.813252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:50.814186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:44:50.814186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:44:50.814186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:44:50.814186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:44:50.814186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:44:50.814186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:44:50.814186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:50.814186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:50.816772 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 313/611] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 361 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:50.818613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7de1bb-02ad-419d-ae3e-3b5dde862e02 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:50.953441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88988500-63c7-458c-9505-b6abad755d8a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:50.968963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:50.999347 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82e4369-4762-4d71-b0cd-1f8ae9625ef0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:51.000342 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82e4369-4762-4d71-b0cd-1f8ae9625ef0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:51.000786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82e4369-4762-4d71-b0cd-1f8ae9625ef0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:51.001298 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82e4369-4762-4d71-b0cd-1f8ae9625ef0 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:51.002086 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82e4369-4762-4d71-b0cd-1f8ae9625ef0 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:51.003810 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82e4369-4762-4d71-b0cd-1f8ae9625ef0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:51.015415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-c82e4369-4762-4d71-b0cd-1f8ae9625ef0 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 16 00:44:51.015415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:51.015415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:44:51.015415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:44:51.015415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:44:51.015415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:44:51.015415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:44:51.015415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:51.015415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 16 00:44:51.015415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:51.015415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:44:51.015415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:51.015415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:51.015415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:51.015415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:51.015415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:51.016668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:51.016668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:51.016668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:51.016668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:51.016668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:51.016668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:51.016668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:51.016668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:44:51.016668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:44:51.016668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:44:51.016668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:44:51.016668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:51.016668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:51.016668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:51.016668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:51.016668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:51.016668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:51.017797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:51.017797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:51.017797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:51.017797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:51.017797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:51.017797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:51.017797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:51.017797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:51.017797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:51.017797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:51.017797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:44:51.017797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:51.017797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 16 00:44:51.017797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:51.020211 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 299/612] 10.222.0.25 () {56 vars in 1132 bytes} [Sun Oct 16 00:44:50 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:51.021316 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82e4369-4762-4d71-b0cd-1f8ae9625ef0 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:51.038173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7de1bb-02ad-419d-ae3e-3b5dde862e02 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:51.091455 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72977067-94d6-4fee-9047-c52cdef2c67e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:51.092445 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72977067-94d6-4fee-9047-c52cdef2c67e None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:51.092897 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72977067-94d6-4fee-9047-c52cdef2c67e None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:51.093345 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72977067-94d6-4fee-9047-c52cdef2c67e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:51.114852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:51.185778 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72977067-94d6-4fee-9047-c52cdef2c67e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:51.188074 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72977067-94d6-4fee-9047-c52cdef2c67e None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:51.216994 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82e4369-4762-4d71-b0cd-1f8ae9625ef0 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:51.282625 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d033d32-819f-4e8a-b221-8c82e047521e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:51.287143 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d033d32-819f-4e8a-b221-8c82e047521e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:51.288018 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d033d32-819f-4e8a-b221-8c82e047521e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:51.288658 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d033d32-819f-4e8a-b221-8c82e047521e None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:51.290255 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d033d32-819f-4e8a-b221-8c82e047521e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:51.290783 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d033d32-819f-4e8a-b221-8c82e047521e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:51.293704 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d033d32-819f-4e8a-b221-8c82e047521e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:51.314316 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 300/613] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:44:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:51.315549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d033d32-819f-4e8a-b221-8c82e047521e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:51.318454 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 314/614] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:44:50 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 501 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 16 00:44:51.320066 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c37a95-f1b3-43e3-9d81-c55d7844f6cf None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:51.320577 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c37a95-f1b3-43e3-9d81-c55d7844f6cf None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:51.320883 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c37a95-f1b3-43e3-9d81-c55d7844f6cf None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:51.322441 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 315/615] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:51.327377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c37a95-f1b3-43e3-9d81-c55d7844f6cf None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:51.364283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:51.395961 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:51.465684 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d033d32-819f-4e8a-b221-8c82e047521e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:51.470195 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c37a95-f1b3-43e3-9d81-c55d7844f6cf None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:51.513274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56019b7-626f-4ca8-920f-06b528175604 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:51.514436 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56019b7-626f-4ca8-920f-06b528175604 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:51.514834 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56019b7-626f-4ca8-920f-06b528175604 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:51.515353 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56019b7-626f-4ca8-920f-06b528175604 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:51.516877 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56019b7-626f-4ca8-920f-06b528175604 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:51.517437 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56019b7-626f-4ca8-920f-06b528175604 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:51.520181 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56019b7-626f-4ca8-920f-06b528175604 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:51.524298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0974de73-abc9-4c6d-be22-e6fd1aae75ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:51.524984 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0974de73-abc9-4c6d-be22-e6fd1aae75ff None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:51.525249 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0974de73-abc9-4c6d-be22-e6fd1aae75ff None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:51.525525 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0974de73-abc9-4c6d-be22-e6fd1aae75ff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:51.534507 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 301/616] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 16 00:44:51 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:51.534507 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56019b7-626f-4ca8-920f-06b528175604 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:51.542514 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0974de73-abc9-4c6d-be22-e6fd1aae75ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:51.545149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0974de73-abc9-4c6d-be22-e6fd1aae75ff None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:51.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-0974de73-abc9-4c6d-be22-e6fd1aae75ff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:51.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:51.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:51.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:51.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:51.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:51.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:51.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:51.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:51.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:51.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:51.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:51.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:51.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:51.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:51.571534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:51.572657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:51.572657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:51.572657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:51.572657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:51.572657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:51.572657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:51.572657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:51.572657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:51.572657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:51.572657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:51.572657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:51.572657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:51.572657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:51.572657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:51.572657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:51.572657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:51.572657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:51.573671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:51.573671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:51.573671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:51.581077 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 316/617] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:51.583327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a74a0a-971f-488a-87f1-88c9b2a77faa None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:51.583793 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a74a0a-971f-488a-87f1-88c9b2a77faa None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:51.584241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a74a0a-971f-488a-87f1-88c9b2a77faa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:51.613417 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-04a74a0a-971f-488a-87f1-88c9b2a77faa None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:51.615601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-04a74a0a-971f-488a-87f1-88c9b2a77faa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:51.713475 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:51.718927 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56019b7-626f-4ca8-920f-06b528175604 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:51.739135 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 317/618] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:51 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:44:51.740567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a74a0a-971f-488a-87f1-88c9b2a77faa None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:51.799520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da77d2f5-c192-4bdb-9849-c5a9455da74d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:51.800729 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da77d2f5-c192-4bdb-9849-c5a9455da74d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:51.801228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da77d2f5-c192-4bdb-9849-c5a9455da74d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:51.801758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da77d2f5-c192-4bdb-9849-c5a9455da74d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:51.803371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da77d2f5-c192-4bdb-9849-c5a9455da74d None admin] RBAC: Adding query filter params (role.id=c5cd777d70a649028f970fd986dc484f, scope.project.id=7deb2c3dae214df7895b40e7339bd6ea, user.id=de64d498da78448394b0d4d1851a2789) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:51.803923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da77d2f5-c192-4bdb-9849-c5a9455da74d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:51.804698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:51.806920 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da77d2f5-c192-4bdb-9849-c5a9455da74d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:51.827564 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 302/619] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:44:51 2022] GET /identity/v3/role_assignments?user.id=de64d498da78448394b0d4d1851a2789&scope.project.id=7deb2c3dae214df7895b40e7339bd6ea&role.id=c5cd777d70a649028f970fd986dc484f => generated 608 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:51.829243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da77d2f5-c192-4bdb-9849-c5a9455da74d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:52.012033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a74a0a-971f-488a-87f1-88c9b2a77faa None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:52.099366 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f57cd64-5af1-4ba6-9182-e08a3d40ac95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:52.100464 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f57cd64-5af1-4ba6-9182-e08a3d40ac95 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:52.101084 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f57cd64-5af1-4ba6-9182-e08a3d40ac95 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:52.103335 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f57cd64-5af1-4ba6-9182-e08a3d40ac95 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:52.104586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f57cd64-5af1-4ba6-9182-e08a3d40ac95 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:52.105127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f57cd64-5af1-4ba6-9182-e08a3d40ac95 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:52.108376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f57cd64-5af1-4ba6-9182-e08a3d40ac95 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:52.120040 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 318/620] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 16 00:44:51 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 380 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:52.173800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da77d2f5-c192-4bdb-9849-c5a9455da74d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:52.239782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19816522-16b9-4179-ac64-63c5a6dfbe4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:52.240740 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19816522-16b9-4179-ac64-63c5a6dfbe4d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:52.241192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19816522-16b9-4179-ac64-63c5a6dfbe4d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:52.241643 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19816522-16b9-4179-ac64-63c5a6dfbe4d None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:52.243317 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19816522-16b9-4179-ac64-63c5a6dfbe4d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:52.243792 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19816522-16b9-4179-ac64-63c5a6dfbe4d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:52.246143 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19816522-16b9-4179-ac64-63c5a6dfbe4d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:52.266749 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f57cd64-5af1-4ba6-9182-e08a3d40ac95 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:52.268563 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 303/621] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 440 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:52.276036 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19816522-16b9-4179-ac64-63c5a6dfbe4d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:52.329649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:52.529581 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f57cd64-5af1-4ba6-9182-e08a3d40ac95 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:52.535472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:52.550244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19816522-16b9-4179-ac64-63c5a6dfbe4d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:52.633777 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c7aaf6-a938-4399-a06e-130b1f20c6b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:52.637799 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c7aaf6-a938-4399-a06e-130b1f20c6b0 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:52.640737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c7aaf6-a938-4399-a06e-130b1f20c6b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:52.642795 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c7aaf6-a938-4399-a06e-130b1f20c6b0 None admin] PATH_INFO: `/v3/services/network` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:52.644232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c7aaf6-a938-4399-a06e-130b1f20c6b0 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:52.649945 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c7aaf6-a938-4399-a06e-130b1f20c6b0 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:52.657848 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2261117d-6fbb-4de8-9d3f-d8a4b4125b9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:52.659397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2261117d-6fbb-4de8-9d3f-d8a4b4125b9d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:52.661667 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2261117d-6fbb-4de8-9d3f-d8a4b4125b9d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:52.663828 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2261117d-6fbb-4de8-9d3f-d8a4b4125b9d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:52.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-98c7aaf6-a938-4399-a06e-130b1f20c6b0 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 16 00:44:52.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:52.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:44:52.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:44:52.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:44:52.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:44:52.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:44:52.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:52.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 16 00:44:52.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:52.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:44:52.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:52.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:52.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:52.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:52.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:52.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:52.688997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:52.688997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:52.688997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:52.688997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:52.688997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:52.688997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:52.688997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:44:52.688997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:44:52.688997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:44:52.688997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:44:52.688997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:52.688997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:52.688997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:52.688997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:52.688997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:52.688997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:52.692231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:52.692231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:52.692231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:52.692231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:52.692231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:52.692231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:52.692231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:52.692231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:52.692231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:52.692231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:52.692231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:44:52.692231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:52.692231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 16 00:44:52.692231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:52.692231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 319/622] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:52 2022] GET /identity/v3/services/network => generated 88 bytes in 423 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:52.693825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2261117d-6fbb-4de8-9d3f-d8a4b4125b9d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:52.693825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2261117d-6fbb-4de8-9d3f-d8a4b4125b9d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:52.700823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c7aaf6-a938-4399-a06e-130b1f20c6b0 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:52.743753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-2261117d-6fbb-4de8-9d3f-d8a4b4125b9d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:52.743753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:52.743753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:52.743753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:52.743753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:52.743753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:52.743753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:52.743753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:52.743753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:52.743753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:52.743753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:52.743753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:52.743753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:44:52.743753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:44:52.743753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:44:52.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:44:52.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:52.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:52.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:52.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:52.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:52.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:52.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:52.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:52.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:52.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:52.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:52.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:52.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:52.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:52.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:52.747148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:52.750656 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:44:52.750656 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:44:52.750656 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:44:52.750656 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:44:52.750656 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:44:52.750656 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:44:52.750656 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:52.750656 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:52.758895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 304/623] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 479 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:52.761652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2261117d-6fbb-4de8-9d3f-d8a4b4125b9d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:52.961319 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c7aaf6-a938-4399-a06e-130b1f20c6b0 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:53.039766 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:53.046890 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3300f237-ca65-42f7-a287-91e7f9070eaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:53.046890 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3300f237-ca65-42f7-a287-91e7f9070eaf None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:53.046890 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3300f237-ca65-42f7-a287-91e7f9070eaf None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:53.046890 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3300f237-ca65-42f7-a287-91e7f9070eaf None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:53.047470 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3300f237-ca65-42f7-a287-91e7f9070eaf None admin] RBAC: Adding query filter params (name=network) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:53.047470 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3300f237-ca65-42f7-a287-91e7f9070eaf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:53.047470 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3300f237-ca65-42f7-a287-91e7f9070eaf None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:53.065159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 320/624] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 16 00:44:52 2022] GET /identity/v3/services?name=network => generated 124 bytes in 365 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:53.078797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3300f237-ca65-42f7-a287-91e7f9070eaf None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:53.082905 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2261117d-6fbb-4de8-9d3f-d8a4b4125b9d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:53.159680 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:53.162113 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eda8de1-96f7-4e13-9b50-3ad670ed5859 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:53.163717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eda8de1-96f7-4e13-9b50-3ad670ed5859 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:53.166346 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eda8de1-96f7-4e13-9b50-3ad670ed5859 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:53.167182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eda8de1-96f7-4e13-9b50-3ad670ed5859 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:53.168723 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eda8de1-96f7-4e13-9b50-3ad670ed5859 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:53.169512 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eda8de1-96f7-4e13-9b50-3ad670ed5859 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:53.172511 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eda8de1-96f7-4e13-9b50-3ad670ed5859 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:53.216911 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 305/625] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:44:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 456 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:53.225869 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eda8de1-96f7-4e13-9b50-3ad670ed5859 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:53.337196 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3300f237-ca65-42f7-a287-91e7f9070eaf None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:53.420092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a785fa3-a651-4310-b638-43b037d5a6b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:53.421552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a785fa3-a651-4310-b638-43b037d5a6b9 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:53.422347 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a785fa3-a651-4310-b638-43b037d5a6b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:53.423105 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a785fa3-a651-4310-b638-43b037d5a6b9 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:53.424329 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a785fa3-a651-4310-b638-43b037d5a6b9 None admin] RBAC: Adding query filter params (type=network) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:53.425659 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a785fa3-a651-4310-b638-43b037d5a6b9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:53.435447 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a785fa3-a651-4310-b638-43b037d5a6b9 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:53.456980 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 321/626] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 16 00:44:53 2022] GET /identity/v3/services?type=network => generated 350 bytes in 375 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:53.463514 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a785fa3-a651-4310-b638-43b037d5a6b9 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:53.591803 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eda8de1-96f7-4e13-9b50-3ad670ed5859 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:53.677024 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca134ae-0229-49fc-aa44-cf4058eb56b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:53.678050 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca134ae-0229-49fc-aa44-cf4058eb56b8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:53.678513 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca134ae-0229-49fc-aa44-cf4058eb56b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:53.679036 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca134ae-0229-49fc-aa44-cf4058eb56b8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:53.680908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca134ae-0229-49fc-aa44-cf4058eb56b8 None admin] RBAC: Adding query filter params (role.id=bfe57c41a4da4b25b9b3e4761f220a46, scope.project.id=7deb2c3dae214df7895b40e7339bd6ea, user.id=33b136afbf8a4d98a9d62053b68707e9) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:53.681460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca134ae-0229-49fc-aa44-cf4058eb56b8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:53.684165 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca134ae-0229-49fc-aa44-cf4058eb56b8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:53.704663 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 306/627] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:44:53 2022] GET /identity/v3/role_assignments?user.id=33b136afbf8a4d98a9d62053b68707e9&scope.project.id=7deb2c3dae214df7895b40e7339bd6ea&role.id=bfe57c41a4da4b25b9b3e4761f220a46 => generated 608 bytes in 480 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:53.759461 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:53.788781 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a785fa3-a651-4310-b638-43b037d5a6b9 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:53.880359 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaedffec-15a2-4d9b-83b0-09677b70671f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:53.881880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaedffec-15a2-4d9b-83b0-09677b70671f None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:53.882803 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaedffec-15a2-4d9b-83b0-09677b70671f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:53.883622 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaedffec-15a2-4d9b-83b0-09677b70671f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:53.885198 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaedffec-15a2-4d9b-83b0-09677b70671f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:53.891460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaedffec-15a2-4d9b-83b0-09677b70671f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:53.938585 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 322/628] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:44:53 2022] POST /identity/v3/endpoints => generated 337 bytes in 476 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:44:54.356692 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c58a72-aebd-4eef-9fe1-d4938996b83f None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:54.357183 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c58a72-aebd-4eef-9fe1-d4938996b83f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:54.357647 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c58a72-aebd-4eef-9fe1-d4938996b83f None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:54.360640 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 307/629] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:54 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:54.366498 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe80b00-d0c1-4d9a-b239-9375fc91b591 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:54.368076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe80b00-d0c1-4d9a-b239-9375fc91b591 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:54.368552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe80b00-d0c1-4d9a-b239-9375fc91b591 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:54.393868 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-dfe80b00-d0c1-4d9a-b239-9375fc91b591 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:54.395457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-dfe80b00-d0c1-4d9a-b239-9375fc91b591 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:54.516079 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 323/630] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:54 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:44:54.794524 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c58a72-aebd-4eef-9fe1-d4938996b83f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:54.809612 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:54.966576 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c58a72-aebd-4eef-9fe1-d4938996b83f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:55.035551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c63cde-8ac2-4b56-b3d4-97ae21967880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:55.036629 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c63cde-8ac2-4b56-b3d4-97ae21967880 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:55.037121 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c63cde-8ac2-4b56-b3d4-97ae21967880 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:55.037614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c63cde-8ac2-4b56-b3d4-97ae21967880 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:55.038648 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c63cde-8ac2-4b56-b3d4-97ae21967880 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:55.041201 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c63cde-8ac2-4b56-b3d4-97ae21967880 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:55.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-f3c63cde-8ac2-4b56-b3d4-97ae21967880 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 16 00:44:55.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:55.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:44:55.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:44:55.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:44:55.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:44:55.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:44:55.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:55.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 16 00:44:55.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:55.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:44:55.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:55.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:55.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:55.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:55.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:55.059689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:55.060721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:55.060721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:55.060721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:55.060721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:55.060721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:55.060721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:55.060721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:44:55.060721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:44:55.060721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:44:55.060721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:44:55.060721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:55.060721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:55.060721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:55.060721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:55.060721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:55.060721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:55.061581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:55.061581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:55.061581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:55.061581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:55.061581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:55.061581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:55.061581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:55.061581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:55.061581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:55.061581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:55.061581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:44:55.061581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:55.061581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 16 00:44:55.061581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:55.062322 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 308/631] 10.222.0.25 () {56 vars in 1129 bytes} [Sun Oct 16 00:44:54 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:55.069222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe80b00-d0c1-4d9a-b239-9375fc91b591 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:55.190069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:55.248132 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe80b00-d0c1-4d9a-b239-9375fc91b591 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:55.315778 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760d3bf2-302a-4d9e-a6df-c64d205b074a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:55.316682 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760d3bf2-302a-4d9e-a6df-c64d205b074a None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:55.317145 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760d3bf2-302a-4d9e-a6df-c64d205b074a None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:55.317613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760d3bf2-302a-4d9e-a6df-c64d205b074a None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:55.318797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760d3bf2-302a-4d9e-a6df-c64d205b074a None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:55.319398 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760d3bf2-302a-4d9e-a6df-c64d205b074a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:55.322181 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760d3bf2-302a-4d9e-a6df-c64d205b074a None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:55.331959 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 324/632] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Oct 16 00:44:55 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:55.339244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c63cde-8ac2-4b56-b3d4-97ae21967880 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:55.505087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:55.528664 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c63cde-8ac2-4b56-b3d4-97ae21967880 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:55.596562 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c4f06d-5af2-4bde-a2b6-92939cdda342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:55.597586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c4f06d-5af2-4bde-a2b6-92939cdda342 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:55.598039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c4f06d-5af2-4bde-a2b6-92939cdda342 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:55.598539 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c4f06d-5af2-4bde-a2b6-92939cdda342 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:55.599794 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c4f06d-5af2-4bde-a2b6-92939cdda342 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:55.600366 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c4f06d-5af2-4bde-a2b6-92939cdda342 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:55.604257 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c4f06d-5af2-4bde-a2b6-92939cdda342 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:55.614570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 309/633] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Oct 16 00:44:55 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:55.621152 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760d3bf2-302a-4d9e-a6df-c64d205b074a None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:55.716521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c0c6ad-b1c2-45ee-a94d-239a7db0a926 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:55.717157 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c0c6ad-b1c2-45ee-a94d-239a7db0a926 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:55.717654 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c0c6ad-b1c2-45ee-a94d-239a7db0a926 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:55.719653 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 310/634] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:55.725471 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3372af38-48c2-42a3-90be-b8530efe8b06 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:55.726058 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3372af38-48c2-42a3-90be-b8530efe8b06 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:55.727074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3372af38-48c2-42a3-90be-b8530efe8b06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:55.748256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3372af38-48c2-42a3-90be-b8530efe8b06 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:55.750013 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3372af38-48c2-42a3-90be-b8530efe8b06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:55.835286 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760d3bf2-302a-4d9e-a6df-c64d205b074a None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:55.843306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 311/635] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:55 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:44:55.877954 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:55.887146 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b818980c-36f7-4c36-becc-adf5cf9c0b41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:55.888158 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b818980c-36f7-4c36-becc-adf5cf9c0b41 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:55.888674 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b818980c-36f7-4c36-becc-adf5cf9c0b41 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:55.889185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b818980c-36f7-4c36-becc-adf5cf9c0b41 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:55.890097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b818980c-36f7-4c36-becc-adf5cf9c0b41 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:55.892832 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b818980c-36f7-4c36-becc-adf5cf9c0b41 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:55.928108 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 325/636] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:44:55 2022] POST /identity/v3/endpoints => generated 346 bytes in 308 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:44:56.037766 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b2e96c-445a-4058-9173-cd8b665ce000 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:56.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b2e96c-445a-4058-9173-cd8b665ce000 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:56.038339 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b2e96c-445a-4058-9173-cd8b665ce000 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:56.041088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 312/637] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:56.047052 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f174e226-fdff-446a-88ab-eaa2537e2c46 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:56.047350 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f174e226-fdff-446a-88ab-eaa2537e2c46 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:56.047622 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f174e226-fdff-446a-88ab-eaa2537e2c46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:56.068625 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-f174e226-fdff-446a-88ab-eaa2537e2c46 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:56.070701 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-f174e226-fdff-446a-88ab-eaa2537e2c46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:56.159831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 326/638] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:56 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:44:56.223971 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b2e96c-445a-4058-9173-cd8b665ce000 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:56.301063 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:56.330889 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40db20b7-054a-4475-b1a1-32900630adba None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:56.331222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40db20b7-054a-4475-b1a1-32900630adba None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:56.331457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40db20b7-054a-4475-b1a1-32900630adba None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:56.333097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 327/639] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:44:56.339431 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dbaeb7-3355-4c9d-a8c0-3a6468e1c94a None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:56.339714 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dbaeb7-3355-4c9d-a8c0-3a6468e1c94a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:56.339993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dbaeb7-3355-4c9d-a8c0-3a6468e1c94a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:56.342801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b2e96c-445a-4058-9173-cd8b665ce000 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:56.364159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-29dbaeb7-3355-4c9d-a8c0-3a6468e1c94a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:56.365393 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-29dbaeb7-3355-4c9d-a8c0-3a6468e1c94a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:56.380859 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99249133-50a5-42a5-b260-0a468f7f3686 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:56.381514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99249133-50a5-42a5-b260-0a468f7f3686 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:56.381796 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99249133-50a5-42a5-b260-0a468f7f3686 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:56.382069 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99249133-50a5-42a5-b260-0a468f7f3686 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:56.383302 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99249133-50a5-42a5-b260-0a468f7f3686 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:56.384691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99249133-50a5-42a5-b260-0a468f7f3686 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:56.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-99249133-50a5-42a5-b260-0a468f7f3686 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 16 00:44:56.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:56.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:44:56.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:44:56.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:44:56.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:44:56.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:44:56.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:56.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 16 00:44:56.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:56.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:44:56.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:56.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:56.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:56.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:56.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:56.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:56.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:56.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:56.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:56.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:56.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:56.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:56.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:44:56.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:44:56.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:44:56.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:44:56.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:56.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:56.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:56.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:56.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:56.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:56.396037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:56.396037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:56.396037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:56.396037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:56.396037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:56.396037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:56.396037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:56.396037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:56.396037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:56.396037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:56.396037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:44:56.396037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:56.396037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 16 00:44:56.396037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:56.396037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 313/640] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 16 00:44:56 2022] GET /identity/v3/services/object-store => generated 93 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:56.401680 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99249133-50a5-42a5-b260-0a468f7f3686 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:56.467101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 328/641] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:56 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:44:56.470367 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29dbaeb7-3355-4c9d-a8c0-3a6468e1c94a None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:56.526798 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99249133-50a5-42a5-b260-0a468f7f3686 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:56.563196 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:56.587015 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:56.604334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efd8f03-4262-4ecd-8211-94a6464070c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:56.605863 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efd8f03-4262-4ecd-8211-94a6464070c5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:56.606617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efd8f03-4262-4ecd-8211-94a6464070c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:56.608200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efd8f03-4262-4ecd-8211-94a6464070c5 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:56.609138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efd8f03-4262-4ecd-8211-94a6464070c5 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:56.610187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efd8f03-4262-4ecd-8211-94a6464070c5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:56.612820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efd8f03-4262-4ecd-8211-94a6464070c5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:56.621425 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 314/642] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 16 00:44:56 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:56.626724 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efd8f03-4262-4ecd-8211-94a6464070c5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:56.641231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29dbaeb7-3355-4c9d-a8c0-3a6468e1c94a None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:56.691271 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1749ba0-f2c8-4985-a8a4-4fb45cd6c350 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:56.692360 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1749ba0-f2c8-4985-a8a4-4fb45cd6c350 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:56.693046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1749ba0-f2c8-4985-a8a4-4fb45cd6c350 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:56.693630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1749ba0-f2c8-4985-a8a4-4fb45cd6c350 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:56.694960 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1749ba0-f2c8-4985-a8a4-4fb45cd6c350 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:56.698233 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1749ba0-f2c8-4985-a8a4-4fb45cd6c350 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:56.734911 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 329/643] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 16 00:44:56 2022] POST /identity/v3/services => generated 269 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:44:56.738650 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1749ba0-f2c8-4985-a8a4-4fb45cd6c350 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:56.772153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efd8f03-4262-4ecd-8211-94a6464070c5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:56.834997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc2870b-707f-41ff-9b5d-69dd19645a02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:56.837240 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc2870b-707f-41ff-9b5d-69dd19645a02 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:56.837951 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc2870b-707f-41ff-9b5d-69dd19645a02 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:56.838431 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc2870b-707f-41ff-9b5d-69dd19645a02 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:56.839401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc2870b-707f-41ff-9b5d-69dd19645a02 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:56.839951 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc2870b-707f-41ff-9b5d-69dd19645a02 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:56.841716 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc2870b-707f-41ff-9b5d-69dd19645a02 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:56.849264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 315/644] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 16 00:44:56 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:56.895106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1749ba0-f2c8-4985-a8a4-4fb45cd6c350 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:56.919291 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:56.969447 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af3f193-b2e1-4392-9033-349ce6d8adb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:56.970544 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af3f193-b2e1-4392-9033-349ce6d8adb0 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:56.971050 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af3f193-b2e1-4392-9033-349ce6d8adb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:56.971535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af3f193-b2e1-4392-9033-349ce6d8adb0 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:56.992083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af3f193-b2e1-4392-9033-349ce6d8adb0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:44:57.002044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-4af3f193-b2e1-4392-9033-349ce6d8adb0 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 16 00:44:57.002044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:57.002044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:57.002044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:57.002044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:57.002044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:57.002044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:57.002044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:57.002044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:57.002044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:57.002044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:57.002044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:57.002044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:44:57.002044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:44:57.002044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:44:57.002044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:44:57.003162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:44:57.003162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:44:57.003162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:44:57.003162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:44:57.003162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:57.003162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:57.003162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:57.003162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:57.003162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:57.003162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:57.003162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:57.003162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:57.003162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:57.003162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:57.003162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:57.003162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:57.004552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:57.004552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:57.004552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:57.004552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:57.004552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:44:57.004552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:44:57.004552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:44:57.004552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:44:57.004552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:44:57.004552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:44:57.004552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 16 00:44:57.004552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:57.005233 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 330/645] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Oct 16 00:44:56 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:57.014693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc2870b-707f-41ff-9b5d-69dd19645a02 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:57.104572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:57.167838 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc2870b-707f-41ff-9b5d-69dd19645a02 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:57.208663 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109425c1-77b0-4c1f-95c1-54c39943aca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:57.209892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109425c1-77b0-4c1f-95c1-54c39943aca4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:57.210180 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109425c1-77b0-4c1f-95c1-54c39943aca4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:57.210443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109425c1-77b0-4c1f-95c1-54c39943aca4 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:57.211334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109425c1-77b0-4c1f-95c1-54c39943aca4 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:57.211751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109425c1-77b0-4c1f-95c1-54c39943aca4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:57.213205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109425c1-77b0-4c1f-95c1-54c39943aca4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:57.223390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 316/646] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Oct 16 00:44:57 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:57.232959 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af3f193-b2e1-4392-9033-349ce6d8adb0 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:57.443398 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af3f193-b2e1-4392-9033-349ce6d8adb0 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:57.510560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036bf17c-9926-4996-841f-c2e8f2c8c894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:57.511569 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036bf17c-9926-4996-841f-c2e8f2c8c894 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:57.512066 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036bf17c-9926-4996-841f-c2e8f2c8c894 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:57.512554 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036bf17c-9926-4996-841f-c2e8f2c8c894 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:57.533594 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036bf17c-9926-4996-841f-c2e8f2c8c894 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:57.536385 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036bf17c-9926-4996-841f-c2e8f2c8c894 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:57.564914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-036bf17c-9926-4996-841f-c2e8f2c8c894 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:57.564914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:57.564914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:57.564914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:57.564914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:57.564914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:57.564914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:57.564914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:57.564914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:57.564914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:57.564914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:57.564914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:57.564914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:57.564914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:57.564914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:57.564914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:57.566770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:57.566770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:57.566770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:57.566770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:57.566770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:57.566770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:57.566770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:57.566770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:57.566770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:57.566770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:57.566770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:57.566770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:57.566770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:57.566770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:57.566770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:57.566770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:57.566770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:57.568549 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:57.568549 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:57.568549 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:57.571095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 331/647] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 339 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:57.580142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109425c1-77b0-4c1f-95c1-54c39943aca4 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:57.752087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109425c1-77b0-4c1f-95c1-54c39943aca4 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:57.771273 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:57.792426 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163b5c7e-5ba4-4893-8507-6ef32bf0363c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:57.793029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163b5c7e-5ba4-4893-8507-6ef32bf0363c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:57.793312 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163b5c7e-5ba4-4893-8507-6ef32bf0363c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:57.793711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163b5c7e-5ba4-4893-8507-6ef32bf0363c None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:57.794385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163b5c7e-5ba4-4893-8507-6ef32bf0363c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:57.794684 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163b5c7e-5ba4-4893-8507-6ef32bf0363c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:57.796227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163b5c7e-5ba4-4893-8507-6ef32bf0363c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:57.811484 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 317/648] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:44:57.821147 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036bf17c-9926-4996-841f-c2e8f2c8c894 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:57.859268 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:58.032093 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036bf17c-9926-4996-841f-c2e8f2c8c894 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:58.104913 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b12d2d7-edc2-46b7-b00a-e42fbc9fd3e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:58.106077 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b12d2d7-edc2-46b7-b00a-e42fbc9fd3e9 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:58.106575 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b12d2d7-edc2-46b7-b00a-e42fbc9fd3e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:58.107080 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b12d2d7-edc2-46b7-b00a-e42fbc9fd3e9 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:58.123489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b12d2d7-edc2-46b7-b00a-e42fbc9fd3e9 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:58.126422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b12d2d7-edc2-46b7-b00a-e42fbc9fd3e9 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:58.150978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-5b12d2d7-edc2-46b7-b00a-e42fbc9fd3e9 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 16 00:44:58.150978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:58.150978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:58.150978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:58.150978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:58.150978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:58.150978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:58.150978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:58.150978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:58.150978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:58.150978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:58.150978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:58.150978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:44:58.150978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:44:58.150978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:44:58.150978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:58.153099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:58.154994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:58.154994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:58.154994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:44:58.154994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:44:58.154994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:44:58.154994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:44:58.154994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:44:58.154994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:44:58.154994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 16 00:44:58.154994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:58.154994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 332/649] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 16 00:44:57 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 334 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:58.162813 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163b5c7e-5ba4-4893-8507-6ef32bf0363c None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:58.316818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163b5c7e-5ba4-4893-8507-6ef32bf0363c None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:58.342181 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b1e2ab-8cb7-4708-9e9d-69a24207e252 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:58.342448 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b1e2ab-8cb7-4708-9e9d-69a24207e252 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:58.342812 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b1e2ab-8cb7-4708-9e9d-69a24207e252 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:58.345894 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 333/650] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:58.350651 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1131555f-1638-4199-9949-d73fb3d27874 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:58.351081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1131555f-1638-4199-9949-d73fb3d27874 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:58.351786 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1131555f-1638-4199-9949-d73fb3d27874 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:58.360564 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b134c2-2f27-4e57-add7-f497b779653e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:58.361198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b134c2-2f27-4e57-add7-f497b779653e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:58.361480 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b134c2-2f27-4e57-add7-f497b779653e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:58.361748 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b134c2-2f27-4e57-add7-f497b779653e None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:58.362485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b134c2-2f27-4e57-add7-f497b779653e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:58.362793 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b134c2-2f27-4e57-add7-f497b779653e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:58.364344 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b134c2-2f27-4e57-add7-f497b779653e None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:58.374932 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-1131555f-1638-4199-9949-d73fb3d27874 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:58.376555 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-1131555f-1638-4199-9949-d73fb3d27874 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:58.385176 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 318/651] 10.222.0.25 () {56 vars in 1149 bytes} [Sun Oct 16 00:44:58 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:58.396082 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b134c2-2f27-4e57-add7-f497b779653e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:58.474806 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:58.483221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 334/652] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:58 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:44:58.597195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b134c2-2f27-4e57-add7-f497b779653e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:58.657413 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f8d978-7f3c-422a-bd9d-b2052f28c9d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:58.658433 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f8d978-7f3c-422a-bd9d-b2052f28c9d3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:58.658945 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f8d978-7f3c-422a-bd9d-b2052f28c9d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:58.659430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f8d978-7f3c-422a-bd9d-b2052f28c9d3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:58.677669 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f8d978-7f3c-422a-bd9d-b2052f28c9d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:58.680495 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f8d978-7f3c-422a-bd9d-b2052f28c9d3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:58.709173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-98f8d978-7f3c-422a-bd9d-b2052f28c9d3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:58.709173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:58.709173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:58.709173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:58.709173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:58.709173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:58.709173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:58.709173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:58.709173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:58.709173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:58.709173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:58.709173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:58.709173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:44:58.709173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:44:58.709173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:44:58.709173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:44:58.711825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:58.711825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:58.711825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:58.711825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:58.711825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:58.711825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:58.711825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:58.711825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:58.711825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:58.711825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:58.711825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:58.711825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:58.711825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:58.711825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:58.711825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:58.711825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:58.711825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:44:58.716126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:44:58.716126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:44:58.716126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:58.716126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 319/653] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:44:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:44:58.721078 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1131555f-1638-4199-9949-d73fb3d27874 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:58.736847 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:58.896202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1131555f-1638-4199-9949-d73fb3d27874 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:58.945827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f8d978-7f3c-422a-bd9d-b2052f28c9d3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:58.958539 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7266d853-3d63-4335-9abb-33a79e60e43f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:58.959543 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7266d853-3d63-4335-9abb-33a79e60e43f None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:58.960019 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7266d853-3d63-4335-9abb-33a79e60e43f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:58.961359 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7266d853-3d63-4335-9abb-33a79e60e43f None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:58.962250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7266d853-3d63-4335-9abb-33a79e60e43f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:58.962775 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7266d853-3d63-4335-9abb-33a79e60e43f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:58.964360 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7266d853-3d63-4335-9abb-33a79e60e43f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:58.984161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 335/654] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:44:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:58.991219 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7266d853-3d63-4335-9abb-33a79e60e43f None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:59.100449 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:59.115869 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:59.143062 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f8d978-7f3c-422a-bd9d-b2052f28c9d3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:59.160469 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7266d853-3d63-4335-9abb-33a79e60e43f None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:59.208394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc184e8e-2c86-4ff6-b669-f7c2fdd398e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:59.209053 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc184e8e-2c86-4ff6-b669-f7c2fdd398e4 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:59.209334 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc184e8e-2c86-4ff6-b669-f7c2fdd398e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:59.209612 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc184e8e-2c86-4ff6-b669-f7c2fdd398e4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:59.218390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc184e8e-2c86-4ff6-b669-f7c2fdd398e4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:59.220103 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc184e8e-2c86-4ff6-b669-f7c2fdd398e4 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:59.231606 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423aff8c-6274-4be6-a4d0-4dfe7f229a5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:59.233555 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423aff8c-6274-4be6-a4d0-4dfe7f229a5c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:59.234261 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423aff8c-6274-4be6-a4d0-4dfe7f229a5c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:59.234432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-dc184e8e-2c86-4ff6-b669-f7c2fdd398e4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:59.234432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:59.234432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:59.234432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:59.234432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:59.234432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:59.234432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:59.234432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:59.234432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:59.234432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:59.234432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:59.234432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:59.234432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:44:59.234432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:44:59.234432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:44:59.236377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:44:59.236377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:59.236377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:59.236377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:59.236377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:59.236377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:59.236377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:59.236377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:59.236377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:59.236377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:59.236377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:59.236377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:59.236377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:59.236377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:59.236377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:59.236377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:59.236377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:59.238326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:44:59.238326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:44:59.238326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:44:59.238326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:44:59.238326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:44:59.238326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:44:59.238326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:44:59.238326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:44:59.239962 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423aff8c-6274-4be6-a4d0-4dfe7f229a5c None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:59.240733 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 336/655] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:44:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:59.241821 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc184e8e-2c86-4ff6-b669-f7c2fdd398e4 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:59.242289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423aff8c-6274-4be6-a4d0-4dfe7f229a5c None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:59.245977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423aff8c-6274-4be6-a4d0-4dfe7f229a5c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:59.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-423aff8c-6274-4be6-a4d0-4dfe7f229a5c None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 16 00:44:59.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:59.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:44:59.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:44:59.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:44:59.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:44:59.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:44:59.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:59.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 16 00:44:59.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:59.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:44:59.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:59.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:44:59.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:44:59.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:44:59.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:44:59.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:44:59.260320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:44:59.260320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:44:59.260320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:44:59.260320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:44:59.260320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:44:59.260320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:44:59.260320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:44:59.260320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:44:59.260320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:44:59.260320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:44:59.260320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:44:59.260320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:44:59.260320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:44:59.260320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:44:59.260320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:44:59.260320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:44:59.261997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:44:59.261997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:44:59.261997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:44:59.261997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:44:59.261997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:44:59.261997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:44:59.261997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:44:59.261997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:44:59.261997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:44:59.261997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:44:59.261997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:44:59.261997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:44:59.261997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 16 00:44:59.261997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:44:59.261997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 320/656] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 16 00:44:58 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:44:59.268918 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423aff8c-6274-4be6-a4d0-4dfe7f229a5c None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:59.442674 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc184e8e-2c86-4ff6-b669-f7c2fdd398e4 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:59.447466 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423aff8c-6274-4be6-a4d0-4dfe7f229a5c None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:59.476967 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:59.489894 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ad0cc6-a37d-4e3f-a809-f44504864d03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:59.490516 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ad0cc6-a37d-4e3f-a809-f44504864d03 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:59.490837 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ad0cc6-a37d-4e3f-a809-f44504864d03 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:59.491131 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ad0cc6-a37d-4e3f-a809-f44504864d03 None admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:59.492003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ad0cc6-a37d-4e3f-a809-f44504864d03 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:59.492347 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ad0cc6-a37d-4e3f-a809-f44504864d03 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:59.493714 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ad0cc6-a37d-4e3f-a809-f44504864d03 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:59.494049 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929bb36e-477e-4215-a988-48a25e023359 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:59.494820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929bb36e-477e-4215-a988-48a25e023359 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:59.495270 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929bb36e-477e-4215-a988-48a25e023359 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:59.495728 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929bb36e-477e-4215-a988-48a25e023359 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:59.496515 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929bb36e-477e-4215-a988-48a25e023359 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:59.496882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929bb36e-477e-4215-a988-48a25e023359 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:59.498665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929bb36e-477e-4215-a988-48a25e023359 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:59.504545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 321/657] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 16 00:44:59 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:59.506680 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88b2a4a-c05d-4ce1-a37b-63d585a40ce3 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:59.507797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88b2a4a-c05d-4ce1-a37b-63d585a40ce3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:59.508263 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88b2a4a-c05d-4ce1-a37b-63d585a40ce3 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:59.510922 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 322/658] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:59 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:59.515712 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88b2a4a-c05d-4ce1-a37b-63d585a40ce3 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:59.517380 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 337/659] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:44:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:59.519252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753fb4b5-8e3c-47b6-9eab-a926b1fb694a None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:59.519252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753fb4b5-8e3c-47b6-9eab-a926b1fb694a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:59.519252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753fb4b5-8e3c-47b6-9eab-a926b1fb694a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:59.538101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-753fb4b5-8e3c-47b6-9eab-a926b1fb694a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:59.539461 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-753fb4b5-8e3c-47b6-9eab-a926b1fb694a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:59.547618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:59.644418 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 338/660] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:59 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:44:59.651183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753fb4b5-8e3c-47b6-9eab-a926b1fb694a None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:44:59.676518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88b2a4a-c05d-4ce1-a37b-63d585a40ce3 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:59.738969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0876046e-6afe-45df-b84b-ef3875f9c7db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:59.740400 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0876046e-6afe-45df-b84b-ef3875f9c7db None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:59.740768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0876046e-6afe-45df-b84b-ef3875f9c7db None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:59.741099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0876046e-6afe-45df-b84b-ef3875f9c7db None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:59.742748 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0876046e-6afe-45df-b84b-ef3875f9c7db None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:44:59.743159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0876046e-6afe-45df-b84b-ef3875f9c7db None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:59.745153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0876046e-6afe-45df-b84b-ef3875f9c7db None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:59.761669 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 323/661] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 16 00:44:59 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:44:59.858010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753fb4b5-8e3c-47b6-9eab-a926b1fb694a None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:44:59.864069 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca1447a-85de-4f6f-9ad5-ffcdb6b0aaab None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:59.865124 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca1447a-85de-4f6f-9ad5-ffcdb6b0aaab None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:59.865901 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca1447a-85de-4f6f-9ad5-ffcdb6b0aaab None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:59.870072 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 324/662] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:44:59 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:44:59.880238 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75ea1e8-53f1-4acc-8296-ff8b1c76230d None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:59.881187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75ea1e8-53f1-4acc-8296-ff8b1c76230d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:59.881961 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75ea1e8-53f1-4acc-8296-ff8b1c76230d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:59.888077 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:44:59.904202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5e699f-e94a-455b-8107-c508dc81c13a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:44:59.905124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5e699f-e94a-455b-8107-c508dc81c13a None admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:44:59.905274 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5e699f-e94a-455b-8107-c508dc81c13a None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:44:59.905423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5e699f-e94a-455b-8107-c508dc81c13a None admin] PATH_INFO: `/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/33b136afbf8a4d98a9d62053b68707e9/roles/7c38bf8daea94adcaaffae01c4253da9` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:44:59.907838 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5e699f-e94a-455b-8107-c508dc81c13a None admin] RBAC: Authorizing `identity:create_grant(project_id=7deb2c3dae214df7895b40e7339bd6ea, user_id=33b136afbf8a4d98a9d62053b68707e9, role_id=7c38bf8daea94adcaaffae01c4253da9)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:44:59.909379 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5e699f-e94a-455b-8107-c508dc81c13a None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:44:59.920233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d75ea1e8-53f1-4acc-8296-ff8b1c76230d None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:44:59.922745 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d75ea1e8-53f1-4acc-8296-ff8b1c76230d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:44:59.930519 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 339/663] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:44:59 2022] PUT /identity/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/33b136afbf8a4d98a9d62053b68707e9/roles/7c38bf8daea94adcaaffae01c4253da9 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:44:59.936117 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5e699f-e94a-455b-8107-c508dc81c13a None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:00.042502 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 325/664] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:44:59 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:00.121113 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5e699f-e94a-455b-8107-c508dc81c13a None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:00.167313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3daac1-4f1a-4f4e-94a5-7cc0656ea6c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:00.167950 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3daac1-4f1a-4f4e-94a5-7cc0656ea6c0 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:00.168235 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3daac1-4f1a-4f4e-94a5-7cc0656ea6c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:00.168592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3daac1-4f1a-4f4e-94a5-7cc0656ea6c0 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:00.169268 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3daac1-4f1a-4f4e-94a5-7cc0656ea6c0 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:00.170664 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3daac1-4f1a-4f4e-94a5-7cc0656ea6c0 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:00.190702 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 340/665] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 16 00:44:59 2022] POST /identity/v3/services => generated 241 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:45:00.328120 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75ea1e8-53f1-4acc-8296-ff8b1c76230d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:00.340865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:00.451866 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75ea1e8-53f1-4acc-8296-ff8b1c76230d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:00.499446 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197aec4a-e3db-4edb-bf6e-9a58205202c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:00.500100 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197aec4a-e3db-4edb-bf6e-9a58205202c8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:00.500383 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197aec4a-e3db-4edb-bf6e-9a58205202c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:00.500651 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197aec4a-e3db-4edb-bf6e-9a58205202c8 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:00.501230 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197aec4a-e3db-4edb-bf6e-9a58205202c8 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:00.502688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197aec4a-e3db-4edb-bf6e-9a58205202c8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:00.512444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-197aec4a-e3db-4edb-bf6e-9a58205202c8 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 16 00:45:00.512444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:00.512444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:45:00.512444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:45:00.512444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:45:00.512444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:45:00.512444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:45:00.512444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:00.512444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 16 00:45:00.512444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:00.512444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:00.512444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:00.512444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:00.512444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:00.512444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:00.512444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:00.514535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:00.514535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:00.514535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:00.514535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:00.514535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:00.514535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:00.514535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:00.514535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:45:00.514535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:45:00.514535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:45:00.514535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:45:00.514535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:00.514535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:00.514535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:00.514535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:00.514535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:00.514535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:00.518381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:00.518381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:00.518381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:00.518381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:00.518381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:00.518381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:00.518381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:00.518381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:00.518381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:00.518381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:00.518381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:45:00.518381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:00.518381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 16 00:45:00.518381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:00.518381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 326/666] 10.222.0.25 () {56 vars in 1132 bytes} [Sun Oct 16 00:45:00 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:45:00.521881 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3daac1-4f1a-4f4e-94a5-7cc0656ea6c0 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:00.574432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:00.652903 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3daac1-4f1a-4f4e-94a5-7cc0656ea6c0 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:00.698342 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e84b72-a2d4-4550-bd4c-0cd0bbd7c78b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:00.699932 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e84b72-a2d4-4550-bd4c-0cd0bbd7c78b None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:00.700876 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e84b72-a2d4-4550-bd4c-0cd0bbd7c78b None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:00.701735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e84b72-a2d4-4550-bd4c-0cd0bbd7c78b None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:00.703294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e84b72-a2d4-4550-bd4c-0cd0bbd7c78b None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:00.704088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e84b72-a2d4-4550-bd4c-0cd0bbd7c78b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:00.706898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e84b72-a2d4-4550-bd4c-0cd0bbd7c78b None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:00.716443 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 341/667] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 16 00:45:00 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:00.727160 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197aec4a-e3db-4edb-bf6e-9a58205202c8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:00.850275 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:00.903999 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197aec4a-e3db-4edb-bf6e-9a58205202c8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:00.951225 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc555689-ecef-4fcd-8798-91217d6f2a19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:00.952121 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc555689-ecef-4fcd-8798-91217d6f2a19 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:00.952582 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc555689-ecef-4fcd-8798-91217d6f2a19 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:00.953000 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc555689-ecef-4fcd-8798-91217d6f2a19 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:00.953838 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc555689-ecef-4fcd-8798-91217d6f2a19 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:00.954321 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc555689-ecef-4fcd-8798-91217d6f2a19 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:00.956223 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc555689-ecef-4fcd-8798-91217d6f2a19 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:00.963045 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 327/668] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 16 00:45:00 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:00.971687 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e84b72-a2d4-4550-bd4c-0cd0bbd7c78b None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:01.105138 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:01.113491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e84b72-a2d4-4550-bd4c-0cd0bbd7c78b None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:01.163938 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27411ea4-424a-4528-8707-be210f8d435c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:01.164382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27411ea4-424a-4528-8707-be210f8d435c None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:01.164948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27411ea4-424a-4528-8707-be210f8d435c None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:01.165106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27411ea4-424a-4528-8707-be210f8d435c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:01.165915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27411ea4-424a-4528-8707-be210f8d435c None admin] RBAC: Adding query filter params (interface=public, service_id=ec52e12be86648a79553c4f489bbbf45, region_id=RegionOne) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:01.166206 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27411ea4-424a-4528-8707-be210f8d435c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:01.167691 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27411ea4-424a-4528-8707-be210f8d435c None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:01.176788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 342/669] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Oct 16 00:45:00 2022] GET /identity/v3/endpoints?service_id=ec52e12be86648a79553c4f489bbbf45&interface=public®ion_id=RegionOne => generated 194 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:01.185622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc555689-ecef-4fcd-8798-91217d6f2a19 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:01.370184 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc555689-ecef-4fcd-8798-91217d6f2a19 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:01.431503 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:01.435448 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fa67f7-b117-4cfd-8d0c-f2f810e172d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:01.436510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fa67f7-b117-4cfd-8d0c-f2f810e172d5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:01.437008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fa67f7-b117-4cfd-8d0c-f2f810e172d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:01.437499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fa67f7-b117-4cfd-8d0c-f2f810e172d5 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:01.438318 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fa67f7-b117-4cfd-8d0c-f2f810e172d5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:01.440377 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fa67f7-b117-4cfd-8d0c-f2f810e172d5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:01.448171 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 328/670] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 16 00:45:01 2022] GET /identity/v3/services => generated 1258 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:45:02.259772 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3952584c-a810-4802-91d8-0aea5ef05f6b None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:02.260342 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3952584c-a810-4802-91d8-0aea5ef05f6b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:02.260342 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3952584c-a810-4802-91d8-0aea5ef05f6b None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:02.261935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 343/671] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:02.267946 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5294525-d645-4906-be46-8f4809b849ce None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:02.268255 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5294525-d645-4906-be46-8f4809b849ce None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:02.268503 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5294525-d645-4906-be46-8f4809b849ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:02.290005 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c5294525-d645-4906-be46-8f4809b849ce None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:02.291263 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c5294525-d645-4906-be46-8f4809b849ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:02.375055 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 329/672] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:02 2022] POST /identity/v3/auth/tokens => generated 1373 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:02.644121 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab87a83-e43c-4e72-9a07-6a9c0bffb172 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:02.644626 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab87a83-e43c-4e72-9a07-6a9c0bffb172 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:02.645123 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab87a83-e43c-4e72-9a07-6a9c0bffb172 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:02.648478 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 344/673] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:02 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:02.648682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5294525-d645-4906-be46-8f4809b849ce None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:02.657316 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a57c382-fcfe-43f9-a45d-3ccdd332365f None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:02.657813 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a57c382-fcfe-43f9-a45d-3ccdd332365f None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:02.658306 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a57c382-fcfe-43f9-a45d-3ccdd332365f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:02.685749 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-1a57c382-fcfe-43f9-a45d-3ccdd332365f None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:02.687036 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-1a57c382-fcfe-43f9-a45d-3ccdd332365f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:02.768623 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 345/674] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:02 2022] POST /identity/v3/auth/tokens => generated 1373 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:02.833199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5294525-d645-4906-be46-8f4809b849ce None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:02.889672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:02.907851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9599f60e-73b7-480a-8e7e-5b8d7145499b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:02.908839 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9599f60e-73b7-480a-8e7e-5b8d7145499b None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:02.909334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9599f60e-73b7-480a-8e7e-5b8d7145499b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:02.909819 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9599f60e-73b7-480a-8e7e-5b8d7145499b None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:02.911096 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9599f60e-73b7-480a-8e7e-5b8d7145499b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:02.913801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9599f60e-73b7-480a-8e7e-5b8d7145499b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:02.942502 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 330/675] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 16 00:45:02 2022] POST /identity/v3/services => generated 251 bytes in 295 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:45:02.984499 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac64403-0e47-4464-9313-48cc92aacbb2 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:02.984850 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac64403-0e47-4464-9313-48cc92aacbb2 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:02.985057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac64403-0e47-4464-9313-48cc92aacbb2 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:02.986809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 346/676] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:02.993031 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd6c6e6-531c-4a97-aba7-ad626e1cb8dc None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:02.993329 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd6c6e6-531c-4a97-aba7-ad626e1cb8dc None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:02.993596 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd6c6e6-531c-4a97-aba7-ad626e1cb8dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:03.016265 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3dd6c6e6-531c-4a97-aba7-ad626e1cb8dc None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:03.017610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3dd6c6e6-531c-4a97-aba7-ad626e1cb8dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:03.080784 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac64403-0e47-4464-9313-48cc92aacbb2 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:03.141324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 331/677] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:02 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:03.277375 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:03.292850 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac64403-0e47-4464-9313-48cc92aacbb2 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:03.363548 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14aae20e-1bfd-425c-ab8c-2c7af79eddf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:03.364653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14aae20e-1bfd-425c-ab8c-2c7af79eddf5 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:03.365160 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14aae20e-1bfd-425c-ab8c-2c7af79eddf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:03.365644 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14aae20e-1bfd-425c-ab8c-2c7af79eddf5 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:03.387674 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14aae20e-1bfd-425c-ab8c-2c7af79eddf5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:45:03.398692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-14aae20e-1bfd-425c-ab8c-2c7af79eddf5 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 16 00:45:03.398692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:03.398692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:03.398692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:03.398692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:03.398692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:03.398692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:03.398692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:03.398692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:03.398692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:03.398692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:03.398692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:03.398692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:45:03.398692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:45:03.398692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:45:03.398692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:45:03.399835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:45:03.399835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:45:03.399835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:45:03.399835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:45:03.399835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:03.399835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:03.399835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:03.399835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:03.399835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:03.399835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:03.399835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:03.399835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:03.399835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:03.399835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:03.399835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:03.399835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:03.400692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:03.400692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:03.400692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:03.400692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:03.400692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:45:03.400692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:45:03.400692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:45:03.400692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:45:03.400692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:45:03.400692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:45:03.400692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 16 00:45:03.400692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:03.402025 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 347/678] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Oct 16 00:45:03 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 321 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:45:03.410996 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd6c6e6-531c-4a97-aba7-ad626e1cb8dc None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:03.450256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14aae20e-1bfd-425c-ab8c-2c7af79eddf5 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:03.600400 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd6c6e6-531c-4a97-aba7-ad626e1cb8dc None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:03.618648 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:03.650359 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fea87aa-5461-4679-9395-686c643a4a93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:03.651396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fea87aa-5461-4679-9395-686c643a4a93 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:03.651930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fea87aa-5461-4679-9395-686c643a4a93 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:03.652432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fea87aa-5461-4679-9395-686c643a4a93 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:03.653409 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fea87aa-5461-4679-9395-686c643a4a93 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:03.653926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fea87aa-5461-4679-9395-686c643a4a93 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:03.655694 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fea87aa-5461-4679-9395-686c643a4a93 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:03.667147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 332/679] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Oct 16 00:45:03 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:03.676064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fea87aa-5461-4679-9395-686c643a4a93 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:03.677299 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14aae20e-1bfd-425c-ab8c-2c7af79eddf5 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:03.717396 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3236eb04-faae-4ea1-8f46-b188f935f3cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:03.718521 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3236eb04-faae-4ea1-8f46-b188f935f3cc None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:03.719297 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3236eb04-faae-4ea1-8f46-b188f935f3cc None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:03.720086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3236eb04-faae-4ea1-8f46-b188f935f3cc None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:03.721459 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3236eb04-faae-4ea1-8f46-b188f935f3cc None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:03.724460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3236eb04-faae-4ea1-8f46-b188f935f3cc None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:03.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-3236eb04-faae-4ea1-8f46-b188f935f3cc None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 16 00:45:03.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:03.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:45:03.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:45:03.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:45:03.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:45:03.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:45:03.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:03.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 16 00:45:03.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:03.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:03.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:03.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:03.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:03.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:03.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:03.737005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:03.738220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:03.738220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:03.738220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:03.738220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:03.738220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:03.738220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:03.738220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:45:03.738220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:45:03.738220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:45:03.738220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:45:03.738220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:03.738220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:03.738220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:03.738220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:03.738220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:03.738220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:03.739161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:03.739161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:03.739161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:03.739161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:03.739161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:03.739161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:03.739161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:03.739161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:03.739161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:03.739161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:03.739161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:45:03.739161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:03.739161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 16 00:45:03.739161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:03.747535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 348/680] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 16 00:45:03 2022] GET /identity/v3/services/object-store => generated 93 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:03.748926 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3236eb04-faae-4ea1-8f46-b188f935f3cc None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:03.783516 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:03.874398 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3236eb04-faae-4ea1-8f46-b188f935f3cc None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:03.890152 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fea87aa-5461-4679-9395-686c643a4a93 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:03.919041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ecb97b-5372-4074-aa28-442a04a2aa82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:03.920597 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ecb97b-5372-4074-aa28-442a04a2aa82 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:03.921095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ecb97b-5372-4074-aa28-442a04a2aa82 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:03.921585 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ecb97b-5372-4074-aa28-442a04a2aa82 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:03.922785 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ecb97b-5372-4074-aa28-442a04a2aa82 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:03.923305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ecb97b-5372-4074-aa28-442a04a2aa82 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:03.925921 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ecb97b-5372-4074-aa28-442a04a2aa82 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:03.941768 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 349/681] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 16 00:45:03 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:03.943979 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ecb97b-5372-4074-aa28-442a04a2aa82 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:03.957393 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eff3a62-c1f3-4ff7-b6d2-25ef7104b3cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:03.958847 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eff3a62-c1f3-4ff7-b6d2-25ef7104b3cc None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:03.959591 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eff3a62-c1f3-4ff7-b6d2-25ef7104b3cc None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:03.960134 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eff3a62-c1f3-4ff7-b6d2-25ef7104b3cc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:03.975148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eff3a62-c1f3-4ff7-b6d2-25ef7104b3cc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:03.977962 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eff3a62-c1f3-4ff7-b6d2-25ef7104b3cc None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:03.997456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-1eff3a62-c1f3-4ff7-b6d2-25ef7104b3cc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:45:03.997456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:03.997456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:03.997456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:03.997456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:03.997456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:03.997456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:03.997456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:03.997456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:03.997456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:03.997456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:03.997456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:03.997456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:45:03.997456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:45:03.997456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:45:03.997456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:45:03.998548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:03.998548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:03.998548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:03.998548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:03.998548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:03.998548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:03.998548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:03.998548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:03.998548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:03.998548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:03.998548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:03.998548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:03.998548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:03.998548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:03.998548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:03.998548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:03.998548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:45:03.999584 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:45:03.999584 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:45:03.999584 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:04.005983 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 333/682] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:45:04.009316 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eff3a62-c1f3-4ff7-b6d2-25ef7104b3cc None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:04.045458 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:04.132958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:04.149578 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eff3a62-c1f3-4ff7-b6d2-25ef7104b3cc None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:04.182061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ecb97b-5372-4074-aa28-442a04a2aa82 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:04.217706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5be8b3d-32f9-40c3-82b2-6b6bc298e0c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:04.219080 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5be8b3d-32f9-40c3-82b2-6b6bc298e0c0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:04.219771 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5be8b3d-32f9-40c3-82b2-6b6bc298e0c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:04.220599 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5be8b3d-32f9-40c3-82b2-6b6bc298e0c0 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:04.222069 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5be8b3d-32f9-40c3-82b2-6b6bc298e0c0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:04.223023 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5be8b3d-32f9-40c3-82b2-6b6bc298e0c0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:04.226655 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5be8b3d-32f9-40c3-82b2-6b6bc298e0c0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:04.229787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050a2a6c-706f-40a1-9934-268275e645d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:04.230283 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050a2a6c-706f-40a1-9934-268275e645d7 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:04.230567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050a2a6c-706f-40a1-9934-268275e645d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:04.230829 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050a2a6c-706f-40a1-9934-268275e645d7 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:04.231514 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050a2a6c-706f-40a1-9934-268275e645d7 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:04.231822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050a2a6c-706f-40a1-9934-268275e645d7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:04.233877 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050a2a6c-706f-40a1-9934-268275e645d7 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:04.244239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 350/683] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 16 00:45:03 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:04.246176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050a2a6c-706f-40a1-9934-268275e645d7 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:04.253586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 334/684] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:45:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:04.264570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5be8b3d-32f9-40c3-82b2-6b6bc298e0c0 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:04.391185 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:04.415789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5be8b3d-32f9-40c3-82b2-6b6bc298e0c0 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:04.416800 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050a2a6c-706f-40a1-9934-268275e645d7 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:04.469994 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6162ef8-ac42-4c82-87c8-b261734e1889 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:04.470549 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf58df8b-4c28-4b5b-9e6a-9e05f5504b15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:04.471043 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6162ef8-ac42-4c82-87c8-b261734e1889 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:04.471499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6162ef8-ac42-4c82-87c8-b261734e1889 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:04.471622 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf58df8b-4c28-4b5b-9e6a-9e05f5504b15 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:04.471948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf58df8b-4c28-4b5b-9e6a-9e05f5504b15 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:04.472047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6162ef8-ac42-4c82-87c8-b261734e1889 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:04.472384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf58df8b-4c28-4b5b-9e6a-9e05f5504b15 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:04.473799 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf58df8b-4c28-4b5b-9e6a-9e05f5504b15 None admin] RBAC: Adding query filter params (interface=public, service_id=e5f4c291a55840f3a89b767e41205881, region_id=RegionOne) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:04.474295 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf58df8b-4c28-4b5b-9e6a-9e05f5504b15 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:04.477713 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf58df8b-4c28-4b5b-9e6a-9e05f5504b15 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:04.486219 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6162ef8-ac42-4c82-87c8-b261734e1889 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:04.486438 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 351/685] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Oct 16 00:45:04 2022] GET /identity/v3/endpoints?service_id=e5f4c291a55840f3a89b767e41205881&interface=public®ion_id=RegionOne => generated 194 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:04.488366 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc79970-8d09-4118-814b-3b78fd03b280 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:04.488868 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc79970-8d09-4118-814b-3b78fd03b280 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:04.488973 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6162ef8-ac42-4c82-87c8-b261734e1889 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:04.489430 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc79970-8d09-4118-814b-3b78fd03b280 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:04.490951 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 352/686] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:04.495264 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc79970-8d09-4118-814b-3b78fd03b280 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:04.509741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-f6162ef8-ac42-4c82-87c8-b261734e1889 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 16 00:45:04.509741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:04.509741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:04.509741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:04.509741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:04.509741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:04.509741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:04.509741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:04.509741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:04.509741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:04.509741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:04.509741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:04.509741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:45:04.509741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:45:04.509741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:45:04.509741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:04.510800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:04.512202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:04.512202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:04.512202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:45:04.512202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:45:04.512202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:45:04.512202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:45:04.512202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:45:04.512202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:45:04.512202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 16 00:45:04.512202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:04.512202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 335/687] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 16 00:45:04 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:04.512202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c796d09-8bc4-4593-9ac5-479a557b37fb None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:04.513283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c796d09-8bc4-4593-9ac5-479a557b37fb None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:04.513283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c796d09-8bc4-4593-9ac5-479a557b37fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:04.521164 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:04.532639 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-9c796d09-8bc4-4593-9ac5-479a557b37fb None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:04.533861 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9c796d09-8bc4-4593-9ac5-479a557b37fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:04.635333 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc79970-8d09-4118-814b-3b78fd03b280 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:04.639815 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 336/688] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:04 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:04.639957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c796d09-8bc4-4593-9ac5-479a557b37fb None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:04.709986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa5cd25-6550-43c3-a3ab-9aa7b6affdab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:04.712224 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa5cd25-6550-43c3-a3ab-9aa7b6affdab None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:04.713069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa5cd25-6550-43c3-a3ab-9aa7b6affdab None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:04.713849 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa5cd25-6550-43c3-a3ab-9aa7b6affdab None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:04.715260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa5cd25-6550-43c3-a3ab-9aa7b6affdab None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:04.717939 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa5cd25-6550-43c3-a3ab-9aa7b6affdab None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:04.731113 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 353/689] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 16 00:45:04 2022] GET /identity/v3/services => generated 1497 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:45:04.799532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:04.852999 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c796d09-8bc4-4593-9ac5-479a557b37fb None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:04.925025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b56edb5-9756-4c6f-b82d-69772eb3733d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:04.926006 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b56edb5-9756-4c6f-b82d-69772eb3733d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:04.926421 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b56edb5-9756-4c6f-b82d-69772eb3733d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:04.926832 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b56edb5-9756-4c6f-b82d-69772eb3733d None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:04.928081 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b56edb5-9756-4c6f-b82d-69772eb3733d None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:04.928537 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b56edb5-9756-4c6f-b82d-69772eb3733d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:04.931964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b56edb5-9756-4c6f-b82d-69772eb3733d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:04.967286 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 337/690] 10.222.0.25 () {56 vars in 1149 bytes} [Sun Oct 16 00:45:04 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:04.974712 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa5cd25-6550-43c3-a3ab-9aa7b6affdab None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:05.120938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b56edb5-9756-4c6f-b82d-69772eb3733d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:05.132267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:05.174452 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa5cd25-6550-43c3-a3ab-9aa7b6affdab None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:05.224746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7207f3-6c59-4dca-a209-a5486aaabb92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:05.225291 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7207f3-6c59-4dca-a209-a5486aaabb92 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:05.225570 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7207f3-6c59-4dca-a209-a5486aaabb92 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:05.226363 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7207f3-6c59-4dca-a209-a5486aaabb92 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:05.268956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b56edb5-9756-4c6f-b82d-69772eb3733d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:05.316542 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2e7973-f816-485b-95cd-b1f478c5b8f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:05.317551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2e7973-f816-485b-95cd-b1f478c5b8f2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:05.318042 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2e7973-f816-485b-95cd-b1f478c5b8f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:05.318744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2e7973-f816-485b-95cd-b1f478c5b8f2 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:05.320138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2e7973-f816-485b-95cd-b1f478c5b8f2 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:05.322283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2e7973-f816-485b-95cd-b1f478c5b8f2 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:05.325387 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:05.339011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-0c2e7973-f816-485b-95cd-b1f478c5b8f2 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 16 00:45:05.339011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:05.339011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:45:05.339011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:45:05.339011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:45:05.339011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:45:05.339011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:45:05.339011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:05.339011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 16 00:45:05.339011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:05.339011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:05.339011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:05.339011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:05.339011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:05.339011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:05.339011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:05.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:05.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:05.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:05.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:05.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:05.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:05.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:05.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:45:05.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:45:05.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:45:05.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:45:05.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:05.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:05.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:05.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:05.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:05.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:05.341085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:05.341085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:05.341085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:05.341085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:05.341085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:05.341085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:05.341085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:05.341085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:05.341085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:05.341085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:05.341085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:45:05.341085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:05.341085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 16 00:45:05.341085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:05.348922 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 338/691] 10.222.0.25 () {56 vars in 1132 bytes} [Sun Oct 16 00:45:05 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:45:05.350688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2e7973-f816-485b-95cd-b1f478c5b8f2 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:05.353363 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7207f3-6c59-4dca-a209-a5486aaabb92 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:05.356564 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7207f3-6c59-4dca-a209-a5486aaabb92 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:05.509489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2e7973-f816-485b-95cd-b1f478c5b8f2 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:05.524777 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:05.565199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27551590-267e-4cad-a1d3-41434a63b832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:05.565821 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27551590-267e-4cad-a1d3-41434a63b832 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:05.566108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27551590-267e-4cad-a1d3-41434a63b832 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:05.566377 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27551590-267e-4cad-a1d3-41434a63b832 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:05.567050 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27551590-267e-4cad-a1d3-41434a63b832 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:05.567361 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27551590-267e-4cad-a1d3-41434a63b832 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:05.568766 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27551590-267e-4cad-a1d3-41434a63b832 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:05.574917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 339/692] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 16 00:45:05 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:05.584663 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27551590-267e-4cad-a1d3-41434a63b832 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:05.603087 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 354/693] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:45:04 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 623 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 16 00:45:05.607120 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7207f3-6c59-4dca-a209-a5486aaabb92 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:05.720718 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:05.811646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27551590-267e-4cad-a1d3-41434a63b832 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:05.827378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7207f3-6c59-4dca-a209-a5486aaabb92 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:05.854480 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365ecc7d-737d-4dba-b4dd-df44de73d31b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:05.855164 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365ecc7d-737d-4dba-b4dd-df44de73d31b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:05.855497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365ecc7d-737d-4dba-b4dd-df44de73d31b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:05.855748 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365ecc7d-737d-4dba-b4dd-df44de73d31b None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:05.856422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365ecc7d-737d-4dba-b4dd-df44de73d31b None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:05.856736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365ecc7d-737d-4dba-b4dd-df44de73d31b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:05.858213 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365ecc7d-737d-4dba-b4dd-df44de73d31b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:05.864513 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 340/694] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 16 00:45:05 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:05.867872 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83131a7-2668-4e85-bf0d-5f386de98de1 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:05.867872 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83131a7-2668-4e85-bf0d-5f386de98de1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:05.867872 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83131a7-2668-4e85-bf0d-5f386de98de1 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:05.868598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 341/695] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:05.875540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83131a7-2668-4e85-bf0d-5f386de98de1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:05.890931 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db3dedf-afb2-451e-ace5-506cbaaecd61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:05.891822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db3dedf-afb2-451e-ace5-506cbaaecd61 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:05.892453 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db3dedf-afb2-451e-ace5-506cbaaecd61 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:05.892865 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db3dedf-afb2-451e-ace5-506cbaaecd61 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:05.907507 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db3dedf-afb2-451e-ace5-506cbaaecd61 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:05.909617 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db3dedf-afb2-451e-ace5-506cbaaecd61 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:05.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-3db3dedf-afb2-451e-ace5-506cbaaecd61 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:45:05.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:05.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:05.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:05.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:05.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:05.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:05.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:05.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:05.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:05.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:05.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:05.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:45:05.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:45:05.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:45:05.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:45:05.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:05.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:05.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:05.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:05.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:05.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:05.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:05.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:05.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:05.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:05.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:05.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:05.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:05.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:05.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:05.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:05.932260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:45:05.933997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:45:05.933997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:45:05.933997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:05.933997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 355/696] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:05.933997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b1a486-39ae-4100-8a39-654ef6d63212 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:05.933997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b1a486-39ae-4100-8a39-654ef6d63212 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:05.933997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b1a486-39ae-4100-8a39-654ef6d63212 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:05.954275 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-a6b1a486-39ae-4100-8a39-654ef6d63212 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:05.955776 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-a6b1a486-39ae-4100-8a39-654ef6d63212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:05.980839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:06.023711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83131a7-2668-4e85-bf0d-5f386de98de1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:06.046253 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 356/697] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:05 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:06.047777 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b1a486-39ae-4100-8a39-654ef6d63212 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:06.067731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:06.080895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b31cbea-4b4d-4be1-8d15-f487448f71bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:06.082146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b31cbea-4b4d-4be1-8d15-f487448f71bb None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:06.082604 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b31cbea-4b4d-4be1-8d15-f487448f71bb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:06.083149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b31cbea-4b4d-4be1-8d15-f487448f71bb None admin] PATH_INFO: `/v3/endpoints` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:06.084029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b31cbea-4b4d-4be1-8d15-f487448f71bb None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:06.086582 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b31cbea-4b4d-4be1-8d15-f487448f71bb None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:06.123311 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 342/698] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:45:05 2022] POST /identity/v3/endpoints => generated 347 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:45:06.220904 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b1a486-39ae-4100-8a39-654ef6d63212 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:06.272056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da2f47e-7f89-4a33-9f02-cd66554423bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:06.273207 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da2f47e-7f89-4a33-9f02-cd66554423bb None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:06.273727 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da2f47e-7f89-4a33-9f02-cd66554423bb None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:06.274186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da2f47e-7f89-4a33-9f02-cd66554423bb None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:06.275150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da2f47e-7f89-4a33-9f02-cd66554423bb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:06.275679 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da2f47e-7f89-4a33-9f02-cd66554423bb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:06.277460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da2f47e-7f89-4a33-9f02-cd66554423bb None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:06.292688 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 357/699] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:45:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:06.302044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b31cbea-4b4d-4be1-8d15-f487448f71bb None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:06.418255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da2f47e-7f89-4a33-9f02-cd66554423bb None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:06.435833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:06.498000 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b31cbea-4b4d-4be1-8d15-f487448f71bb None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:06.535523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da2f47e-7f89-4a33-9f02-cd66554423bb None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:06.567805 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7bbff3a-17b2-49f2-9f95-fc14059e4dc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:06.568662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bbff3a-17b2-49f2-9f95-fc14059e4dc7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:06.568816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bbff3a-17b2-49f2-9f95-fc14059e4dc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:06.568952 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bbff3a-17b2-49f2-9f95-fc14059e4dc7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:06.577876 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0e7d9c-c7ef-4224-a241-8913d0d99d9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:06.578522 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e7d9c-c7ef-4224-a241-8913d0d99d9c None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:06.578807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e7d9c-c7ef-4224-a241-8913d0d99d9c None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:06.579111 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0e7d9c-c7ef-4224-a241-8913d0d99d9c None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:06.579688 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0e7d9c-c7ef-4224-a241-8913d0d99d9c None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:06.581325 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0e7d9c-c7ef-4224-a241-8913d0d99d9c None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:06.587965 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7bbff3a-17b2-49f2-9f95-fc14059e4dc7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:06.591239 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7bbff3a-17b2-49f2-9f95-fc14059e4dc7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:06.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-3b0e7d9c-c7ef-4224-a241-8913d0d99d9c None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 16 00:45:06.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:06.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:45:06.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:45:06.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:45:06.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:45:06.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:45:06.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:06.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 16 00:45:06.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:06.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:06.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:06.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:06.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:06.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:06.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:06.591444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:06.593384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:06.593384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:06.593384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:06.593384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:06.593384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:06.593384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:06.593384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:45:06.593384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:45:06.593384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:45:06.593384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:45:06.593384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:06.593384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:06.593384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:06.593384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:06.593384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:06.593384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:06.595189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:06.595189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:06.595189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:06.595189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:06.595189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:06.595189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:06.595189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:06.595189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:06.595189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:06.595189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:06.595189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:45:06.595189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:06.595189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 16 00:45:06.595189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:06.597930 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 358/700] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 16 00:45:06 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:45:06.598846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0e7d9c-c7ef-4224-a241-8913d0d99d9c None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:06.632469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-d7bbff3a-17b2-49f2-9f95-fc14059e4dc7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:45:06.632469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:06.632469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:06.632469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:06.632469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:06.632469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:06.632469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:06.632469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:06.632469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:06.632469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:06.632469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:06.632469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:06.632469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:45:06.632469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:45:06.632469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:45:06.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:45:06.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:06.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:06.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:06.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:06.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:06.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:06.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:06.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:06.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:06.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:06.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:06.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:06.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:06.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:06.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:06.634610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:06.636430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:45:06.636430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:45:06.636430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:45:06.636430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:45:06.636430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:45:06.636430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:45:06.636430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:45:06.636430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:06.637312 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 343/701] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:45:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 336 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:06.650592 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7bbff3a-17b2-49f2-9f95-fc14059e4dc7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:06.717798 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:06.749393 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0e7d9c-c7ef-4224-a241-8913d0d99d9c None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:06.778326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:06.823236 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cad5245-64a2-450d-be4f-acb30ff0a31c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:06.824642 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cad5245-64a2-450d-be4f-acb30ff0a31c None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:06.825250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cad5245-64a2-450d-be4f-acb30ff0a31c None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:06.825873 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cad5245-64a2-450d-be4f-acb30ff0a31c None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:06.827377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cad5245-64a2-450d-be4f-acb30ff0a31c None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:06.827974 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cad5245-64a2-450d-be4f-acb30ff0a31c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:06.830956 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cad5245-64a2-450d-be4f-acb30ff0a31c None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:06.841346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 359/702] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 16 00:45:06 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:06.846649 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cad5245-64a2-450d-be4f-acb30ff0a31c None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:06.925461 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7bbff3a-17b2-49f2-9f95-fc14059e4dc7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:06.972318 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cad5245-64a2-450d-be4f-acb30ff0a31c None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:07.016200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40b4cc6-4c17-4d99-9dc3-013b738c8374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:07.017064 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40b4cc6-4c17-4d99-9dc3-013b738c8374 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:07.017444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40b4cc6-4c17-4d99-9dc3-013b738c8374 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:07.017857 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40b4cc6-4c17-4d99-9dc3-013b738c8374 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:07.018600 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40b4cc6-4c17-4d99-9dc3-013b738c8374 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:07.019049 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40b4cc6-4c17-4d99-9dc3-013b738c8374 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:07.020699 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40b4cc6-4c17-4d99-9dc3-013b738c8374 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:07.021013 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b62b505-9502-4550-907e-6e96800c0f57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:07.022800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b62b505-9502-4550-907e-6e96800c0f57 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:07.024561 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b62b505-9502-4550-907e-6e96800c0f57 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:07.025615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b62b505-9502-4550-907e-6e96800c0f57 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:07.027264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b62b505-9502-4550-907e-6e96800c0f57 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:07.028039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b62b505-9502-4550-907e-6e96800c0f57 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:07.031058 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b62b505-9502-4550-907e-6e96800c0f57 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:07.032600 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 360/703] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 16 00:45:06 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:07.034067 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40b4cc6-4c17-4d99-9dc3-013b738c8374 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:07.040069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:07.065929 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 344/704] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:45:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 416 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:07.072071 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b62b505-9502-4550-907e-6e96800c0f57 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:07.205029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b62b505-9502-4550-907e-6e96800c0f57 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:07.206684 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:07.243263 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40b4cc6-4c17-4d99-9dc3-013b738c8374 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:07.257787 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f230a483-9977-47a2-86a3-19da88bdf89a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:07.259770 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f230a483-9977-47a2-86a3-19da88bdf89a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:07.260818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f230a483-9977-47a2-86a3-19da88bdf89a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:07.261427 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f230a483-9977-47a2-86a3-19da88bdf89a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:07.263394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f230a483-9977-47a2-86a3-19da88bdf89a None admin] RBAC: Adding query filter params (role.id=7c38bf8daea94adcaaffae01c4253da9, scope.project.id=7deb2c3dae214df7895b40e7339bd6ea, user.id=33b136afbf8a4d98a9d62053b68707e9) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:07.264925 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f230a483-9977-47a2-86a3-19da88bdf89a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:07.269385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f230a483-9977-47a2-86a3-19da88bdf89a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:07.287332 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 345/705] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:45:07 2022] GET /identity/v3/role_assignments?user.id=33b136afbf8a4d98a9d62053b68707e9&scope.project.id=7deb2c3dae214df7895b40e7339bd6ea&role.id=7c38bf8daea94adcaaffae01c4253da9 => generated 608 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:07.296531 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8bd311-8803-40bb-b0d3-16693cefe7b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:07.297262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8bd311-8803-40bb-b0d3-16693cefe7b0 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:07.297553 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8bd311-8803-40bb-b0d3-16693cefe7b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:07.297815 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8bd311-8803-40bb-b0d3-16693cefe7b0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:07.298532 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8bd311-8803-40bb-b0d3-16693cefe7b0 None admin] RBAC: Adding query filter params (interface=public, service_id=c0322813717a48f5bc6154e25efc09f1, region_id=RegionOne) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:07.298840 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8bd311-8803-40bb-b0d3-16693cefe7b0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:07.300256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8bd311-8803-40bb-b0d3-16693cefe7b0 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:07.314911 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 361/706] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Oct 16 00:45:07 2022] GET /identity/v3/endpoints?service_id=c0322813717a48f5bc6154e25efc09f1&interface=public®ion_id=RegionOne => generated 194 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:07.318998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f230a483-9977-47a2-86a3-19da88bdf89a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:07.536131 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f230a483-9977-47a2-86a3-19da88bdf89a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:07.604456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4055dcdb-7bb1-4383-8553-270db8df2db5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:07.606754 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4055dcdb-7bb1-4383-8553-270db8df2db5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:07.607556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4055dcdb-7bb1-4383-8553-270db8df2db5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:07.608307 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4055dcdb-7bb1-4383-8553-270db8df2db5 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:07.609727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4055dcdb-7bb1-4383-8553-270db8df2db5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:07.612716 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4055dcdb-7bb1-4383-8553-270db8df2db5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:07.626317 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 346/707] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 16 00:45:07 2022] GET /identity/v3/services => generated 1497 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:45:07.776230 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedf2522-08cb-49e3-97e3-2e30d5c19a45 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:07.777029 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedf2522-08cb-49e3-97e3-2e30d5c19a45 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:07.777029 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedf2522-08cb-49e3-97e3-2e30d5c19a45 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:07.778502 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 362/708] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:07.785621 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1161dca1-0860-471c-9455-1eaf848aae83 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:07.786122 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1161dca1-0860-471c-9455-1eaf848aae83 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:07.786691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1161dca1-0860-471c-9455-1eaf848aae83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:07.821249 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-1161dca1-0860-471c-9455-1eaf848aae83 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:07.823425 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1161dca1-0860-471c-9455-1eaf848aae83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:07.870962 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:07.954475 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 347/709] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:07 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:08.226505 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedf2522-08cb-49e3-97e3-2e30d5c19a45 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:08.346904 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:08.411955 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedf2522-08cb-49e3-97e3-2e30d5c19a45 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:08.479230 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679f118c-dcd1-421e-ad45-64133626a4ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:08.480221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679f118c-dcd1-421e-ad45-64133626a4ad None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:08.480705 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679f118c-dcd1-421e-ad45-64133626a4ad None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:08.481181 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679f118c-dcd1-421e-ad45-64133626a4ad None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:08.482295 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679f118c-dcd1-421e-ad45-64133626a4ad None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:08.484906 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679f118c-dcd1-421e-ad45-64133626a4ad None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:08.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-679f118c-dcd1-421e-ad45-64133626a4ad None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 16 00:45:08.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:08.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:45:08.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:45:08.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:45:08.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:45:08.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:45:08.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:08.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 16 00:45:08.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:08.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:08.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:08.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:08.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:08.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:08.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:08.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:08.503946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:08.503946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:08.503946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:08.503946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:08.503946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:08.503946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:08.503946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:45:08.503946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:45:08.503946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:45:08.503946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:45:08.503946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:08.503946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:08.503946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:08.503946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:08.503946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:08.503946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:08.504833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:08.504833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:08.504833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:08.504833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:08.504833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:08.504833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:08.504833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:08.504833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:08.504833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:08.504833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:08.504833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:45:08.504833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:08.504833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 16 00:45:08.504833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:08.505630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 363/710] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 16 00:45:08 2022] GET /identity/v3/services/object-store => generated 93 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:08.513282 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1161dca1-0860-471c-9455-1eaf848aae83 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:08.708341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1161dca1-0860-471c-9455-1eaf848aae83 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:08.776533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da410e7-c55b-4423-9c92-08d6499e0b4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:08.777516 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da410e7-c55b-4423-9c92-08d6499e0b4d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:08.777957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da410e7-c55b-4423-9c92-08d6499e0b4d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:08.778435 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da410e7-c55b-4423-9c92-08d6499e0b4d None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:08.779636 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da410e7-c55b-4423-9c92-08d6499e0b4d None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:08.780176 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da410e7-c55b-4423-9c92-08d6499e0b4d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:08.782854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da410e7-c55b-4423-9c92-08d6499e0b4d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:08.794115 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 348/711] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 16 00:45:08 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:08.800743 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679f118c-dcd1-421e-ad45-64133626a4ad None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:08.988703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679f118c-dcd1-421e-ad45-64133626a4ad None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:09.044830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:09.054378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf2d989-b890-4fad-93cf-c32af9e058d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:09.055236 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf2d989-b890-4fad-93cf-c32af9e058d5 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:09.055587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf2d989-b890-4fad-93cf-c32af9e058d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:09.056150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf2d989-b890-4fad-93cf-c32af9e058d5 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:09.057312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf2d989-b890-4fad-93cf-c32af9e058d5 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:09.057874 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf2d989-b890-4fad-93cf-c32af9e058d5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:09.060483 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf2d989-b890-4fad-93cf-c32af9e058d5 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:09.071035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 364/712] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 16 00:45:08 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:09.077180 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da410e7-c55b-4423-9c92-08d6499e0b4d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:09.122916 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:09.232529 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da410e7-c55b-4423-9c92-08d6499e0b4d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:09.274248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600ef667-2117-47c8-9033-29020ecffad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:09.275072 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600ef667-2117-47c8-9033-29020ecffad1 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:09.275477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600ef667-2117-47c8-9033-29020ecffad1 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:09.275850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600ef667-2117-47c8-9033-29020ecffad1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:09.276645 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600ef667-2117-47c8-9033-29020ecffad1 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:09.278142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600ef667-2117-47c8-9033-29020ecffad1 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:09.304957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 349/713] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:45:09 2022] POST /identity/v3/endpoints => generated 349 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:45:09.469171 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5022a3c-2632-4448-8667-654f0a503c14 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:09.469741 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5022a3c-2632-4448-8667-654f0a503c14 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:09.470227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5022a3c-2632-4448-8667-654f0a503c14 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:09.473322 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 365/714] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:09 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:09.483409 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fcf1d1-52fd-4dc6-b3c4-73e32a51bd29 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:09.483977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fcf1d1-52fd-4dc6-b3c4-73e32a51bd29 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:09.484532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fcf1d1-52fd-4dc6-b3c4-73e32a51bd29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:09.519745 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-12fcf1d1-52fd-4dc6-b3c4-73e32a51bd29 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:09.522073 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-12fcf1d1-52fd-4dc6-b3c4-73e32a51bd29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:09.659211 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 350/715] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:09 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:09.954586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5022a3c-2632-4448-8667-654f0a503c14 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:10.091326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5022a3c-2632-4448-8667-654f0a503c14 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:10.136114 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f881dc41-0cfb-4ae0-8818-8eabfba9c3ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:10.136767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f881dc41-0cfb-4ae0-8818-8eabfba9c3ad None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:10.137046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f881dc41-0cfb-4ae0-8818-8eabfba9c3ad None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:10.137323 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f881dc41-0cfb-4ae0-8818-8eabfba9c3ad None admin] PATH_INFO: `/v3/services/compute` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:10.137919 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f881dc41-0cfb-4ae0-8818-8eabfba9c3ad None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:10.139344 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f881dc41-0cfb-4ae0-8818-8eabfba9c3ad None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:10.149573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-f881dc41-0cfb-4ae0-8818-8eabfba9c3ad None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 16 00:45:10.149573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:10.149573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:45:10.149573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:45:10.149573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:45:10.149573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:45:10.149573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:45:10.149573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:10.149573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 16 00:45:10.149573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:10.149573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:10.149573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:10.149573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:10.149573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:10.149573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:10.149573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:10.149573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:10.151523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:10.151523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:10.151523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:10.151523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:10.151523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:10.151523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:10.151523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:45:10.151523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:45:10.151523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:45:10.151523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:45:10.151523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:10.151523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:10.151523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:10.151523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:10.151523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:10.151523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:10.153180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:10.153180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:10.153180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:10.153180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:10.153180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:10.153180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:10.153180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:10.153180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:10.153180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:10.153180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:10.153180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:45:10.153180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:10.153180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 16 00:45:10.153180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:10.153180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 366/716] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:45:09 2022] GET /identity/v3/services/compute => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:45:10.160072 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fcf1d1-52fd-4dc6-b3c4-73e32a51bd29 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:10.170057 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:10.230639 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22ccd73-fa0f-4f00-9d02-59ae8fcb7cba None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:10.230639 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22ccd73-fa0f-4f00-9d02-59ae8fcb7cba None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:10.231062 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22ccd73-fa0f-4f00-9d02-59ae8fcb7cba None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:10.232391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 367/717] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:45:10.238987 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6df383a-3052-451b-b12a-34ac4b2a9abb None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:10.239885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6df383a-3052-451b-b12a-34ac4b2a9abb None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:10.240637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6df383a-3052-451b-b12a-34ac4b2a9abb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:10.277954 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-d6df383a-3052-451b-b12a-34ac4b2a9abb None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:10.279282 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d6df383a-3052-451b-b12a-34ac4b2a9abb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:10.341287 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fcf1d1-52fd-4dc6-b3c4-73e32a51bd29 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:10.357837 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:10.383852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 368/718] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:10 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:45:10.400596 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d37c45-90da-4acb-abe0-bb5b84d78193 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:10.402087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d37c45-90da-4acb-abe0-bb5b84d78193 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:10.402793 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d37c45-90da-4acb-abe0-bb5b84d78193 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:10.403537 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d37c45-90da-4acb-abe0-bb5b84d78193 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:10.405180 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d37c45-90da-4acb-abe0-bb5b84d78193 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:10.405944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d37c45-90da-4acb-abe0-bb5b84d78193 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:10.408887 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d37c45-90da-4acb-abe0-bb5b84d78193 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:10.419421 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 351/719] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 16 00:45:10 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:10.431954 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6df383a-3052-451b-b12a-34ac4b2a9abb None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:10.438852 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f24d670-ba79-4769-ab34-f7adf4a57759 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:10.439390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f24d670-ba79-4769-ab34-f7adf4a57759 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:10.439861 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f24d670-ba79-4769-ab34-f7adf4a57759 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:10.442855 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 352/720] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:10 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:10.449033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6686dbc6-8ab1-4e32-be5d-4e61d8eefd41 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:10.449326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6686dbc6-8ab1-4e32-be5d-4e61d8eefd41 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:10.449627 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6686dbc6-8ab1-4e32-be5d-4e61d8eefd41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:10.470290 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-6686dbc6-8ab1-4e32-be5d-4e61d8eefd41 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:10.471636 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6686dbc6-8ab1-4e32-be5d-4e61d8eefd41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:10.569349 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:10.583119 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 353/721] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:10 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:45:10.626569 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6df383a-3052-451b-b12a-34ac4b2a9abb None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:10.681150 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6686dbc6-8ab1-4e32-be5d-4e61d8eefd41 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:10.699968 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e929b9-2b2b-41ea-9899-fac7f22ff593 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:10.700497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e929b9-2b2b-41ea-9899-fac7f22ff593 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:10.700748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e929b9-2b2b-41ea-9899-fac7f22ff593 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:10.701024 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e929b9-2b2b-41ea-9899-fac7f22ff593 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:10.701944 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e929b9-2b2b-41ea-9899-fac7f22ff593 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:10.702267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e929b9-2b2b-41ea-9899-fac7f22ff593 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:10.704179 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e929b9-2b2b-41ea-9899-fac7f22ff593 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:10.713177 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 369/722] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 16 00:45:10 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:10.846382 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6686dbc6-8ab1-4e32-be5d-4e61d8eefd41 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:10.920009 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c49533-eea3-4dec-9a33-41b877d1f55a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:10.921087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c49533-eea3-4dec-9a33-41b877d1f55a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:10.921584 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c49533-eea3-4dec-9a33-41b877d1f55a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:10.922083 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c49533-eea3-4dec-9a33-41b877d1f55a None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:10.923194 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c49533-eea3-4dec-9a33-41b877d1f55a None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:10.925984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c49533-eea3-4dec-9a33-41b877d1f55a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:10.946657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-f2c49533-eea3-4dec-9a33-41b877d1f55a None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 16 00:45:10.946657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:10.946657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:45:10.946657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:45:10.946657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:45:10.946657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:45:10.946657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:45:10.946657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:10.946657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 16 00:45:10.946657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:10.946657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:10.946657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:10.946657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:10.946657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:10.946657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:10.946657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:10.946657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:10.948699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:10.948699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:10.948699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:10.948699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:10.948699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:10.948699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:10.948699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:45:10.948699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:45:10.948699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:45:10.948699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:45:10.948699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:10.948699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:10.948699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:10.948699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:10.948699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:10.948699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:10.950468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:10.950468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:10.950468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:10.950468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:10.950468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:10.950468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:10.950468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:10.950468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:10.950468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:10.950468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:10.950468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:45:10.950468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:10.950468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 16 00:45:10.950468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:10.950468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 354/723] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 16 00:45:10 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:45:10.958534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e929b9-2b2b-41ea-9899-fac7f22ff593 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:11.011276 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c49533-eea3-4dec-9a33-41b877d1f55a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:11.076977 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:11.107813 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:11.135280 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e929b9-2b2b-41ea-9899-fac7f22ff593 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:11.177029 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19d8877-0959-4262-ada0-9664a044b0e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:11.177664 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19d8877-0959-4262-ada0-9664a044b0e8 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:11.177948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19d8877-0959-4262-ada0-9664a044b0e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:11.178233 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19d8877-0959-4262-ada0-9664a044b0e8 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:11.178940 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19d8877-0959-4262-ada0-9664a044b0e8 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:11.179898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19d8877-0959-4262-ada0-9664a044b0e8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:11.180732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19d8877-0959-4262-ada0-9664a044b0e8 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:11.193395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 370/724] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 16 00:45:10 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:11.195964 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19d8877-0959-4262-ada0-9664a044b0e8 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:11.203631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c49533-eea3-4dec-9a33-41b877d1f55a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:11.272996 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c44d41-53d4-49f3-abc1-38ae165cae24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:11.274000 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c44d41-53d4-49f3-abc1-38ae165cae24 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:11.274478 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c44d41-53d4-49f3-abc1-38ae165cae24 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:11.274994 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c44d41-53d4-49f3-abc1-38ae165cae24 None admin] PATH_INFO: `/v3/services/image` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:11.276058 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c44d41-53d4-49f3-abc1-38ae165cae24 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:11.278687 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c44d41-53d4-49f3-abc1-38ae165cae24 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:11.297428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-c0c44d41-53d4-49f3-abc1-38ae165cae24 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 16 00:45:11.297428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:11.297428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:45:11.297428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:45:11.297428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:45:11.297428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:45:11.297428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:45:11.297428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:11.297428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 16 00:45:11.297428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:11.297428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:11.297428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:11.297428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:11.297428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:11.297428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:11.297428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:11.297428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:11.298543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:11.298543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:11.298543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:11.298543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:11.298543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:11.298543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:11.298543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:45:11.298543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:45:11.298543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:45:11.298543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:45:11.298543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:11.298543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:11.298543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:11.298543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:11.298543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:11.298543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:11.299467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:11.299467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:11.299467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:11.299467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:11.299467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:11.299467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:11.299467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:11.299467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:11.299467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:11.299467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:11.299467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:45:11.299467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:11.299467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 16 00:45:11.299467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:11.306160 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 355/725] 10.222.0.25 () {56 vars in 1105 bytes} [Sun Oct 16 00:45:11 2022] GET /identity/v3/services/image => generated 86 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:45:11.311094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c44d41-53d4-49f3-abc1-38ae165cae24 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:11.347752 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19d8877-0959-4262-ada0-9664a044b0e8 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:11.371611 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:11.395042 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52189928-7800-4264-b670-3d2096dc23e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:11.396109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52189928-7800-4264-b670-3d2096dc23e5 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:11.396613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52189928-7800-4264-b670-3d2096dc23e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:11.397102 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52189928-7800-4264-b670-3d2096dc23e5 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:11.398069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52189928-7800-4264-b670-3d2096dc23e5 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:11.404216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52189928-7800-4264-b670-3d2096dc23e5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:11.406028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52189928-7800-4264-b670-3d2096dc23e5 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:11.417957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 371/726] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 16 00:45:11 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:11.419369 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52189928-7800-4264-b670-3d2096dc23e5 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:11.549209 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c44d41-53d4-49f3-abc1-38ae165cae24 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:11.552589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52189928-7800-4264-b670-3d2096dc23e5 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:11.602010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868f69f6-698a-4d1e-b8ae-2eadb6a8b933 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:11.603328 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868f69f6-698a-4d1e-b8ae-2eadb6a8b933 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:11.603827 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868f69f6-698a-4d1e-b8ae-2eadb6a8b933 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:11.604317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868f69f6-698a-4d1e-b8ae-2eadb6a8b933 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:11.605461 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868f69f6-698a-4d1e-b8ae-2eadb6a8b933 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:11.606956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:11.608166 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868f69f6-698a-4d1e-b8ae-2eadb6a8b933 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:11.626016 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa21832f-fe6e-46ce-90fa-a01b795e131c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:11.627339 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa21832f-fe6e-46ce-90fa-a01b795e131c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:11.627945 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa21832f-fe6e-46ce-90fa-a01b795e131c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:11.628497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa21832f-fe6e-46ce-90fa-a01b795e131c None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:11.630395 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa21832f-fe6e-46ce-90fa-a01b795e131c None admin] RBAC: Adding query filter params (name=image) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:11.630977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa21832f-fe6e-46ce-90fa-a01b795e131c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:11.634149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa21832f-fe6e-46ce-90fa-a01b795e131c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:11.636931 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 372/727] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:45:11 2022] POST /identity/v3/endpoints => generated 346 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:45:11.658922 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 356/728] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:11 2022] GET /identity/v3/services?name=image => generated 122 bytes in 349 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:11.664331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868f69f6-698a-4d1e-b8ae-2eadb6a8b933 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:11.712457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:11.797398 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868f69f6-698a-4d1e-b8ae-2eadb6a8b933 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:11.838687 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d966652-1a3d-4b69-bfa0-c575f1bb692d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:11.839407 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d966652-1a3d-4b69-bfa0-c575f1bb692d None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:11.839697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d966652-1a3d-4b69-bfa0-c575f1bb692d None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:11.840072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d966652-1a3d-4b69-bfa0-c575f1bb692d None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:11.840781 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d966652-1a3d-4b69-bfa0-c575f1bb692d None admin] RBAC: Adding query filter params (type=image) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:11.841166 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d966652-1a3d-4b69-bfa0-c575f1bb692d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:11.842578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d966652-1a3d-4b69-bfa0-c575f1bb692d None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:11.848476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 373/729] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:11 2022] GET /identity/v3/services?type=image => generated 122 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:12.648554 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fd5c58-abbe-443d-8342-057e4c6a1201 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:12.649080 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fd5c58-abbe-443d-8342-057e4c6a1201 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:12.649531 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fd5c58-abbe-443d-8342-057e4c6a1201 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:12.652461 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 357/730] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:12 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:45:12.661994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8788fe02-4ee4-49a5-9363-593b0a31610a None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:12.662512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8788fe02-4ee4-49a5-9363-593b0a31610a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:12.663083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8788fe02-4ee4-49a5-9363-593b0a31610a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:12.695602 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-8788fe02-4ee4-49a5-9363-593b0a31610a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:12.697755 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8788fe02-4ee4-49a5-9363-593b0a31610a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:12.828814 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 374/731] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:12 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:13.329207 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fd5c58-abbe-443d-8342-057e4c6a1201 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:13.472373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fd5c58-abbe-443d-8342-057e4c6a1201 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:13.515009 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec80d94-6d3f-4b24-b00e-4c4f742e2c12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:13.516041 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec80d94-6d3f-4b24-b00e-4c4f742e2c12 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:13.516569 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec80d94-6d3f-4b24-b00e-4c4f742e2c12 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:13.517098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec80d94-6d3f-4b24-b00e-4c4f742e2c12 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:13.517993 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec80d94-6d3f-4b24-b00e-4c4f742e2c12 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:13.519809 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec80d94-6d3f-4b24-b00e-4c4f742e2c12 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:13.530518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-cec80d94-6d3f-4b24-b00e-4c4f742e2c12 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 16 00:45:13.530518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:13.530518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:45:13.530518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:45:13.530518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:45:13.530518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:45:13.530518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:45:13.530518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:13.530518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 16 00:45:13.530518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:13.530518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:13.530518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:13.530518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:13.530518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:13.530518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:13.530518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:13.530518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:13.531807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:13.531807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:13.531807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:13.531807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:13.531807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:13.531807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:13.531807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:45:13.531807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:45:13.531807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:45:13.531807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:45:13.531807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:13.531807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:13.531807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:13.531807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:13.531807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:13.531807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:13.532695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:13.532695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:13.532695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:13.532695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:13.532695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:13.532695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:13.532695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:13.532695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:13.532695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:13.532695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:13.532695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:45:13.532695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:13.532695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 16 00:45:13.532695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:13.535702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 358/732] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 16 00:45:13 2022] GET /identity/v3/services/object-store => generated 93 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:13.545859 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8788fe02-4ee4-49a5-9363-593b0a31610a None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:13.593110 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:13.690484 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8788fe02-4ee4-49a5-9363-593b0a31610a None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:13.732095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079c8efb-36cf-4030-b619-d425409a490d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:13.732825 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079c8efb-36cf-4030-b619-d425409a490d None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:13.733146 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079c8efb-36cf-4030-b619-d425409a490d None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:13.733461 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079c8efb-36cf-4030-b619-d425409a490d None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:13.734186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079c8efb-36cf-4030-b619-d425409a490d None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:13.734499 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079c8efb-36cf-4030-b619-d425409a490d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:13.736094 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079c8efb-36cf-4030-b619-d425409a490d None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:13.742461 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 375/733] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 16 00:45:13 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:13.750971 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec80d94-6d3f-4b24-b00e-4c4f742e2c12 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:13.816822 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:13.879284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec80d94-6d3f-4b24-b00e-4c4f742e2c12 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:13.932613 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969de483-2ff1-4cc6-997c-38b95bd35d3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:13.933303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969de483-2ff1-4cc6-997c-38b95bd35d3c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:13.933605 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969de483-2ff1-4cc6-997c-38b95bd35d3c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:13.933968 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969de483-2ff1-4cc6-997c-38b95bd35d3c None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:13.934682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969de483-2ff1-4cc6-997c-38b95bd35d3c None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:13.935046 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969de483-2ff1-4cc6-997c-38b95bd35d3c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:13.936619 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969de483-2ff1-4cc6-997c-38b95bd35d3c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:13.943025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 359/734] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 16 00:45:13 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:13.949675 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079c8efb-36cf-4030-b619-d425409a490d None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:14.076717 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:14.092423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079c8efb-36cf-4030-b619-d425409a490d None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:14.134898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a527118c-03cf-4d1c-a2bd-39febaa992c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:14.135951 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a527118c-03cf-4d1c-a2bd-39febaa992c0 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:14.136460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a527118c-03cf-4d1c-a2bd-39febaa992c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:14.136947 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a527118c-03cf-4d1c-a2bd-39febaa992c0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:14.137981 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a527118c-03cf-4d1c-a2bd-39febaa992c0 None admin] RBAC: Adding query filter params (interface=admin, service_id=e5f4c291a55840f3a89b767e41205881, region_id=RegionOne) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:14.138552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a527118c-03cf-4d1c-a2bd-39febaa992c0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:14.140304 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a527118c-03cf-4d1c-a2bd-39febaa992c0 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:14.147664 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 376/735] 10.222.0.25 () {56 vars in 1249 bytes} [Sun Oct 16 00:45:13 2022] GET /identity/v3/endpoints?service_id=e5f4c291a55840f3a89b767e41205881&interface=admin®ion_id=RegionOne => generated 193 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:14.154149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969de483-2ff1-4cc6-997c-38b95bd35d3c None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:14.251267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c12c746-eb14-4f72-93c0-d5f505ba3727 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:14.251633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c12c746-eb14-4f72-93c0-d5f505ba3727 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:14.251871 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c12c746-eb14-4f72-93c0-d5f505ba3727 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:14.253217 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 377/736] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:45:14.258977 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfbafc5-268e-4f78-9021-340614514c3d None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:14.259355 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfbafc5-268e-4f78-9021-340614514c3d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:14.259621 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfbafc5-268e-4f78-9021-340614514c3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:14.282695 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-8cfbafc5-268e-4f78-9021-340614514c3d None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:14.284317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8cfbafc5-268e-4f78-9021-340614514c3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:14.287452 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969de483-2ff1-4cc6-997c-38b95bd35d3c None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:14.312490 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:14.329600 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae96220f-201b-4a8f-b6a9-a63b7169fce2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:14.330487 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae96220f-201b-4a8f-b6a9-a63b7169fce2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:14.330772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae96220f-201b-4a8f-b6a9-a63b7169fce2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:14.331143 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae96220f-201b-4a8f-b6a9-a63b7169fce2 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:14.331781 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae96220f-201b-4a8f-b6a9-a63b7169fce2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:14.333161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae96220f-201b-4a8f-b6a9-a63b7169fce2 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:14.343527 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 360/737] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 16 00:45:14 2022] GET /identity/v3/services => generated 1497 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:45:14.343527 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12598977-0f86-40d2-a276-35415101fa54 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:14.343527 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12598977-0f86-40d2-a276-35415101fa54 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:14.343527 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12598977-0f86-40d2-a276-35415101fa54 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:14.344351 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 361/738] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:14.350237 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602f2f72-60e9-4108-8300-814902f5f1de None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:14.350558 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602f2f72-60e9-4108-8300-814902f5f1de None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:14.351023 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602f2f72-60e9-4108-8300-814902f5f1de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:14.372021 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-602f2f72-60e9-4108-8300-814902f5f1de None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:14.373274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-602f2f72-60e9-4108-8300-814902f5f1de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:14.387735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 378/739] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:14 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:45:14.389429 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192c81d0-c576-4432-a8c2-055f7019587a None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:14.389951 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192c81d0-c576-4432-a8c2-055f7019587a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:14.390437 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192c81d0-c576-4432-a8c2-055f7019587a None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:14.392465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 379/740] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:14.401050 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dd1dd5-b4c2-4471-8928-f16e9ef36b1c None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:14.401408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dd1dd5-b4c2-4471-8928-f16e9ef36b1c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:14.401686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dd1dd5-b4c2-4471-8928-f16e9ef36b1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:14.423244 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-c9dd1dd5-b4c2-4471-8928-f16e9ef36b1c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:14.424814 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c9dd1dd5-b4c2-4471-8928-f16e9ef36b1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:14.500046 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 362/741] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:14 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:45:14.528946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 380/742] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:14 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:45:14.787141 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602f2f72-60e9-4108-8300-814902f5f1de None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:14.805376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dd1dd5-b4c2-4471-8928-f16e9ef36b1c None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:14.827125 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:14.924964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602f2f72-60e9-4108-8300-814902f5f1de None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:14.942251 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:14.967228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcb8f02-7e50-4360-9f0e-cf3d70eddeca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:14.967458 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dd1dd5-b4c2-4471-8928-f16e9ef36b1c None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:14.967863 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcb8f02-7e50-4360-9f0e-cf3d70eddeca None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:14.968146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcb8f02-7e50-4360-9f0e-cf3d70eddeca None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:14.968423 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcb8f02-7e50-4360-9f0e-cf3d70eddeca None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:14.969100 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcb8f02-7e50-4360-9f0e-cf3d70eddeca None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:14.970589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcb8f02-7e50-4360-9f0e-cf3d70eddeca None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:14.973317 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: WARNING keystone.server.flask.common [None req-3bcb8f02-7e50-4360-9f0e-cf3d70eddeca None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 16 00:45:14.993203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.sql.core [None req-3bcb8f02-7e50-4360-9f0e-cf3d70eddeca None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 16 00:45:14.993203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 16 00:45:14.993203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [parameters: {'id': '00b809fb3ea64eef9366a0b25c547af2', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 16 00:45:14.993203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81485) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 16 00:45:15.001250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-3bcb8f02-7e50-4360-9f0e-cf3d70eddeca None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 00b809fb3ea64eef9366a0b25c547af2.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 00b809fb3ea64eef9366a0b25c547af2. Oct 16 00:45:15.001250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:15.001250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 16 00:45:15.001250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 16 00:45:15.001250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 16 00:45:15.001250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 16 00:45:15.001250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 16 00:45:15.001250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result = self._query(query) Oct 16 00:45:15.001250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 16 00:45:15.001250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application conn.query(q) Oct 16 00:45:15.001250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 16 00:45:15.001250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 16 00:45:15.001250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result.read() Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 16 00:45:15.002142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 16 00:45:15.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 16 00:45:15.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application next(self.gen) Oct 16 00:45:15.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 16 00:45:15.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application yield resource Oct 16 00:45:15.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 16 00:45:15.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application next(self.gen) Oct 16 00:45:15.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 16 00:45:15.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.session.rollback() Oct 16 00:45:15.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 16 00:45:15.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.force_reraise() Oct 16 00:45:15.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 16 00:45:15.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise self.value Oct 16 00:45:15.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 16 00:45:15.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 16 00:45:15.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 16 00:45:15.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application session.commit() Oct 16 00:45:15.003198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 16 00:45:15.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 16 00:45:15.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 16 00:45:15.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._prepare_impl() Oct 16 00:45:15.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 16 00:45:15.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.session.flush() Oct 16 00:45:15.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 16 00:45:15.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._flush(objects) Oct 16 00:45:15.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 16 00:45:15.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 16 00:45:15.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 16 00:45:15.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application compat.raise_( Oct 16 00:45:15.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 16 00:45:15.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception Oct 16 00:45:15.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 16 00:45:15.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application flush_context.execute() Oct 16 00:45:15.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 16 00:45:15.004137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rec.execute(self) Oct 16 00:45:15.005103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 16 00:45:15.005103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 16 00:45:15.005103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 16 00:45:15.005103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 16 00:45:15.005103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 16 00:45:15.005103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 16 00:45:15.005103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 16 00:45:15.005103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 16 00:45:15.005103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 16 00:45:15.005103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 16 00:45:15.005103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 16 00:45:15.005103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 16 00:45:15.005103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 16 00:45:15.005103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 16 00:45:15.005103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 16 00:45:15.009059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 16 00:45:15.009059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 16 00:45:15.009059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception Oct 16 00:45:15.009059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 16 00:45:15.009059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 16 00:45:15.009059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 16 00:45:15.009059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 16 00:45:15.009059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 16 00:45:15.009059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result = self._query(query) Oct 16 00:45:15.009059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 16 00:45:15.009059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application conn.query(q) Oct 16 00:45:15.009059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 16 00:45:15.009059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 16 00:45:15.009059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 16 00:45:15.009059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application result.read() Oct 16 00:45:15.009059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 16 00:45:15.009059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 16 00:45:15.011693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 16 00:45:15.011693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 16 00:45:15.011693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 16 00:45:15.011693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 16 00:45:15.011693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 16 00:45:15.011693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 16 00:45:15.011693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 16 00:45:15.011693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 16 00:45:15.011693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [parameters: {'id': '00b809fb3ea64eef9366a0b25c547af2', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 16 00:45:15.011693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 16 00:45:15.011693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:15.011693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:15.011693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:15.011693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:15.011693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 16 00:45:15.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 16 00:45:15.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 16 00:45:15.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 16 00:45:15.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 16 00:45:15.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:15.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:15.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:15.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:15.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:15.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:15.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:15.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:15.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:15.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:15.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:15.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:15.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:15.016735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:15.016735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 16 00:45:15.016735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 16 00:45:15.016735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:15.016735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:15.016735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 16 00:45:15.016735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 16 00:45:15.016735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 00b809fb3ea64eef9366a0b25c547af2. Oct 16 00:45:15.016735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:15.016735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 363/743] 10.222.0.25 () {60 vars in 1140 bytes} [Sun Oct 16 00:45:14 2022] POST /identity/v3/projects => generated 262 bytes in 222 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:15.016735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcb8f02-7e50-4360-9f0e-cf3d70eddeca None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:15.033756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ec253d-19af-4c8f-8e66-6cbeba2754de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:15.034880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ec253d-19af-4c8f-8e66-6cbeba2754de None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:15.035423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ec253d-19af-4c8f-8e66-6cbeba2754de None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:15.035855 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ec253d-19af-4c8f-8e66-6cbeba2754de None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:15.036990 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ec253d-19af-4c8f-8e66-6cbeba2754de None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:15.040626 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ec253d-19af-4c8f-8e66-6cbeba2754de None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:15.063078 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 381/744] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 16 00:45:14 2022] POST /identity/v3/services => generated 242 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:45:15.064256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ec253d-19af-4c8f-8e66-6cbeba2754de None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:15.182893 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcb8f02-7e50-4360-9f0e-cf3d70eddeca None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:15.198632 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:15.237425 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80217138-ad11-43fe-b8aa-606fc88fbe06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:15.238517 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80217138-ad11-43fe-b8aa-606fc88fbe06 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:15.239096 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80217138-ad11-43fe-b8aa-606fc88fbe06 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:15.239561 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80217138-ad11-43fe-b8aa-606fc88fbe06 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:15.240468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80217138-ad11-43fe-b8aa-606fc88fbe06 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:15.242359 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80217138-ad11-43fe-b8aa-606fc88fbe06 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:15.252012 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ec253d-19af-4c8f-8e66-6cbeba2754de None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:15.263542 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 364/745] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 16 00:45:15 2022] POST /identity/v3/services => generated 242 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:45:15.326721 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9f3153-a8f1-4852-82ce-f946bd4a3040 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:15.327422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9f3153-a8f1-4852-82ce-f946bd4a3040 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:15.327707 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9f3153-a8f1-4852-82ce-f946bd4a3040 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:15.327986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9f3153-a8f1-4852-82ce-f946bd4a3040 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:15.337569 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9f3153-a8f1-4852-82ce-f946bd4a3040 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:15.339226 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9f3153-a8f1-4852-82ce-f946bd4a3040 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:15.352496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-ea9f3153-a8f1-4852-82ce-f946bd4a3040 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:45:15.352496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:15.352496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:15.352496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:15.352496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:15.352496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:15.352496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:15.352496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:15.352496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:15.352496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:15.352496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:15.352496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:15.352496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:45:15.352496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:45:15.352496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:45:15.353463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:45:15.353463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:15.353463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:15.353463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:15.353463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:15.353463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:15.353463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:15.353463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:15.353463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:15.353463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:15.353463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:15.353463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:15.353463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:15.353463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:15.353463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:15.353463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:15.353463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:15.354376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:45:15.354376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:45:15.354376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:45:15.354376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:45:15.354376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:45:15.354376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:45:15.354376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:45:15.354376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:15.354376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 382/746] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:45:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:15.360021 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80217138-ad11-43fe-b8aa-606fc88fbe06 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:15.393301 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:15.494043 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80217138-ad11-43fe-b8aa-606fc88fbe06 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:15.537522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb55f06-5701-4b9a-add9-da45f9cb1160 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:15.538251 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb55f06-5701-4b9a-add9-da45f9cb1160 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:15.538765 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb55f06-5701-4b9a-add9-da45f9cb1160 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:15.539124 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb55f06-5701-4b9a-add9-da45f9cb1160 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:15.539774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb55f06-5701-4b9a-add9-da45f9cb1160 None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:15.540085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb55f06-5701-4b9a-add9-da45f9cb1160 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:15.541576 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb55f06-5701-4b9a-add9-da45f9cb1160 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:15.561875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 365/747] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 16 00:45:15 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:17.601644 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2169dea6-604d-46b5-8926-b2108875e9c6 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:17.602166 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2169dea6-604d-46b5-8926-b2108875e9c6 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:17.602243 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2169dea6-604d-46b5-8926-b2108875e9c6 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:17.604853 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 383/748] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:17.610517 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745abe2c-61bd-4bf0-b0cb-4bb2a9e025f3 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:17.611450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745abe2c-61bd-4bf0-b0cb-4bb2a9e025f3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:17.611930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745abe2c-61bd-4bf0-b0cb-4bb2a9e025f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:17.633290 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-745abe2c-61bd-4bf0-b0cb-4bb2a9e025f3 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:17.635238 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-745abe2c-61bd-4bf0-b0cb-4bb2a9e025f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:17.648513 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc407c9-75ac-4878-afba-1153ec29eb26 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:17.648922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc407c9-75ac-4878-afba-1153ec29eb26 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:17.649212 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc407c9-75ac-4878-afba-1153ec29eb26 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:17.651333 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 384/749] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:17 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:17.656741 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d1353c-11e6-48bc-b55c-75108ff40896 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:17.657033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d1353c-11e6-48bc-b55c-75108ff40896 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:17.657316 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d1353c-11e6-48bc-b55c-75108ff40896 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:17.680398 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-36d1353c-11e6-48bc-b55c-75108ff40896 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:17.681793 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-36d1353c-11e6-48bc-b55c-75108ff40896 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:17.731650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 366/750] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:17 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:17.766517 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 385/751] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:17 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:17.932633 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dbf334-7055-46ca-a329-5f92045592e9 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:17.932831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dbf334-7055-46ca-a329-5f92045592e9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:17.933113 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dbf334-7055-46ca-a329-5f92045592e9 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:17.934749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 367/752] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:17.940245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbeb988c-717d-4b28-8424-91d474310490 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:17.940535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbeb988c-717d-4b28-8424-91d474310490 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:17.940815 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbeb988c-717d-4b28-8424-91d474310490 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:17.961251 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-cbeb988c-717d-4b28-8424-91d474310490 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:17.962464 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-cbeb988c-717d-4b28-8424-91d474310490 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:18.015164 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dbf334-7055-46ca-a329-5f92045592e9 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:18.029534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:18.041179 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:18.041708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 386/753] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:17 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:18.043123 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de576314-abae-4ce3-8e30-893f2ee1f18b None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:18.043402 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de576314-abae-4ce3-8e30-893f2ee1f18b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:18.043652 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de576314-abae-4ce3-8e30-893f2ee1f18b None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:18.045338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 387/754] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:18.052526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de576314-abae-4ce3-8e30-893f2ee1f18b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:18.138819 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dbf334-7055-46ca-a329-5f92045592e9 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:18.176679 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09ea4df-c10f-43ea-8994-d99f4c3a276b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:18.177335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09ea4df-c10f-43ea-8994-d99f4c3a276b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:18.177669 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09ea4df-c10f-43ea-8994-d99f4c3a276b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:18.177978 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09ea4df-c10f-43ea-8994-d99f4c3a276b None admin] PATH_INFO: `/v3/services/image` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:18.178558 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09ea4df-c10f-43ea-8994-d99f4c3a276b None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:18.180107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09ea4df-c10f-43ea-8994-d99f4c3a276b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:18.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-b09ea4df-c10f-43ea-8994-d99f4c3a276b None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 16 00:45:18.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:18.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:45:18.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:45:18.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:45:18.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:45:18.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:45:18.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:18.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 16 00:45:18.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:18.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:18.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:18.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:18.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:18.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:18.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:18.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:18.191874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:18.191874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:18.191874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:18.191874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:18.191874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:18.191874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:18.191874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:45:18.191874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:45:18.191874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:45:18.191874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:45:18.191874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:18.191874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:18.191874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:18.191874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:18.191874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:18.191874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:18.194683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:18.194683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:18.194683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:18.194683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:18.194683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:18.194683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:18.194683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:18.194683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:18.194683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:18.194683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:18.194683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:45:18.194683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:18.194683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 16 00:45:18.194683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:18.194683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 368/755] 10.222.0.25 () {56 vars in 1105 bytes} [Sun Oct 16 00:45:18 2022] GET /identity/v3/services/image => generated 86 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:45:18.197466 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c232d0-8a43-4532-9ef5-f00b0ffafca3 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:18.197466 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c232d0-8a43-4532-9ef5-f00b0ffafca3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:18.197466 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c232d0-8a43-4532-9ef5-f00b0ffafca3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:18.213427 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-10c232d0-8a43-4532-9ef5-f00b0ffafca3 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:18.214667 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-10c232d0-8a43-4532-9ef5-f00b0ffafca3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:18.283013 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de576314-abae-4ce3-8e30-893f2ee1f18b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:18.297520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 369/756] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:18 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:18.298435 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c232d0-8a43-4532-9ef5-f00b0ffafca3 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:18.316602 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:18.345049 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b64ab44-77df-4e18-82df-843241cc5f78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:18.346045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b64ab44-77df-4e18-82df-843241cc5f78 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:18.346328 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b64ab44-77df-4e18-82df-843241cc5f78 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:18.346755 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b64ab44-77df-4e18-82df-843241cc5f78 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:18.347492 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b64ab44-77df-4e18-82df-843241cc5f78 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:18.348884 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b64ab44-77df-4e18-82df-843241cc5f78 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:18.360591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-3b64ab44-77df-4e18-82df-843241cc5f78 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 16 00:45:18.360591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:18.360591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:45:18.360591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:45:18.360591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:45:18.360591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:45:18.360591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:45:18.360591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:18.360591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 16 00:45:18.360591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:18.360591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:18.360591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:18.360591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:18.360591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:18.360591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:18.360591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:18.360591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:18.361637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:18.361637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:18.361637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:18.361637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:18.361637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:18.361637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:18.361637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:45:18.361637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:45:18.361637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:45:18.361637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:45:18.361637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:18.361637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:18.361637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:18.361637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:18.361637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:18.361637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:18.362493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:18.362493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:18.362493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:18.362493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:18.362493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:18.362493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:18.362493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:18.362493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:18.362493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:18.362493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:18.362493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:45:18.362493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:18.362493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 16 00:45:18.362493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:18.363940 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 388/757] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:45:18 2022] GET /identity/v3/services/compute => generated 88 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:18.364977 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b64ab44-77df-4e18-82df-843241cc5f78 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:18.445172 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c232d0-8a43-4532-9ef5-f00b0ffafca3 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:18.452620 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:18.498113 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2468a84c-67b2-400e-9388-f4c4381431ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:18.499840 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2468a84c-67b2-400e-9388-f4c4381431ca None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:18.500358 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2468a84c-67b2-400e-9388-f4c4381431ca None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:18.500932 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2468a84c-67b2-400e-9388-f4c4381431ca None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:18.502062 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2468a84c-67b2-400e-9388-f4c4381431ca None admin] RBAC: Adding query filter params (name=image) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:18.502567 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2468a84c-67b2-400e-9388-f4c4381431ca None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:18.505302 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2468a84c-67b2-400e-9388-f4c4381431ca None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:18.512697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b64ab44-77df-4e18-82df-843241cc5f78 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:18.519108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 370/758] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:18 2022] GET /identity/v3/services?name=image => generated 122 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:18.519108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2468a84c-67b2-400e-9388-f4c4381431ca None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:18.571538 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3bb19a-1cfa-4f09-81f9-e19760277bf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:18.572512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3bb19a-1cfa-4f09-81f9-e19760277bf3 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:18.572977 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3bb19a-1cfa-4f09-81f9-e19760277bf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:18.573424 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3bb19a-1cfa-4f09-81f9-e19760277bf3 None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:18.574515 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3bb19a-1cfa-4f09-81f9-e19760277bf3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:18.576672 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3bb19a-1cfa-4f09-81f9-e19760277bf3 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:18.581161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: WARNING keystone.server.flask.common [None req-df3bb19a-1cfa-4f09-81f9-e19760277bf3 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 16 00:45:18.598349 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.sql.core [None req-df3bb19a-1cfa-4f09-81f9-e19760277bf3 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 16 00:45:18.598349 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 16 00:45:18.598349 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [parameters: {'user_id': '431aa500d0d94d1d9d45e490ee5a03c9', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 16 00:45:18.598349 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81486) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 16 00:45:18.606528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-df3bb19a-1cfa-4f09-81f9-e19760277bf3 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 16 00:45:18.606528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:18.606528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 16 00:45:18.606528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 16 00:45:18.606528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 16 00:45:18.606528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 16 00:45:18.606528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 16 00:45:18.606528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application result = self._query(query) Oct 16 00:45:18.606528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 16 00:45:18.606528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application conn.query(q) Oct 16 00:45:18.606528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 16 00:45:18.606528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 16 00:45:18.606528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 16 00:45:18.606528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application result.read() Oct 16 00:45:18.606528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 16 00:45:18.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application next(self.gen) Oct 16 00:45:18.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 16 00:45:18.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application yield resource Oct 16 00:45:18.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 16 00:45:18.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application next(self.gen) Oct 16 00:45:18.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 16 00:45:18.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self.session.rollback() Oct 16 00:45:18.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 16 00:45:18.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self.force_reraise() Oct 16 00:45:18.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 16 00:45:18.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise self.value Oct 16 00:45:18.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 16 00:45:18.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 16 00:45:18.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 16 00:45:18.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application session.commit() Oct 16 00:45:18.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 16 00:45:18.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 16 00:45:18.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 16 00:45:18.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._prepare_impl() Oct 16 00:45:18.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 16 00:45:18.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self.session.flush() Oct 16 00:45:18.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 16 00:45:18.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._flush(objects) Oct 16 00:45:18.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 16 00:45:18.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 16 00:45:18.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 16 00:45:18.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application compat.raise_( Oct 16 00:45:18.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 16 00:45:18.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception Oct 16 00:45:18.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 16 00:45:18.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application flush_context.execute() Oct 16 00:45:18.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 16 00:45:18.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rec.execute(self) Oct 16 00:45:18.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 16 00:45:18.617822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 16 00:45:18.619885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 16 00:45:18.619885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 16 00:45:18.619885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 16 00:45:18.619885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 16 00:45:18.619885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 16 00:45:18.619885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 16 00:45:18.619885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 16 00:45:18.619885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 16 00:45:18.619885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 16 00:45:18.619885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 16 00:45:18.619885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 16 00:45:18.619885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 16 00:45:18.619885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 16 00:45:18.619885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 16 00:45:18.619885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 16 00:45:18.619885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception Oct 16 00:45:18.622340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 16 00:45:18.622340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 16 00:45:18.622340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 16 00:45:18.622340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 16 00:45:18.622340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 16 00:45:18.622340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application result = self._query(query) Oct 16 00:45:18.622340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 16 00:45:18.622340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application conn.query(q) Oct 16 00:45:18.622340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 16 00:45:18.622340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 16 00:45:18.622340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 16 00:45:18.622340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application result.read() Oct 16 00:45:18.622340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 16 00:45:18.622340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 16 00:45:18.622340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 16 00:45:18.622340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 16 00:45:18.625294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 16 00:45:18.625294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 16 00:45:18.625294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 16 00:45:18.625294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 16 00:45:18.625294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 16 00:45:18.625294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 16 00:45:18.625294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [parameters: {'user_id': '431aa500d0d94d1d9d45e490ee5a03c9', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 16 00:45:18.625294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 16 00:45:18.625294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:18.625294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:18.625294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:18.625294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:18.625294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:18.625294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:18.625294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:18.627313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:18.627313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:18.627313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:18.627313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:18.627313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:18.627313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:18.627313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:18.627313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 16 00:45:18.627313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 16 00:45:18.627313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:18.627313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:18.627313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:45:18.627313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:45:18.627313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:45:18.627313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:45:18.627313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 16 00:45:18.627313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 16 00:45:18.629395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 16 00:45:18.629395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 16 00:45:18.629395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 16 00:45:18.629395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 16 00:45:18.629395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 16 00:45:18.629395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:18.629395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 389/759] 10.222.0.25 () {60 vars in 1131 bytes} [Sun Oct 16 00:45:18 2022] POST /identity/v3/users => generated 160 bytes in 243 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:18.629395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3bb19a-1cfa-4f09-81f9-e19760277bf3 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:18.633798 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:18.660506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2468a84c-67b2-400e-9388-f4c4381431ca None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:18.697307 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0110f5f-d313-4cb9-aaf5-7d876cecef66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:18.697841 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0110f5f-d313-4cb9-aaf5-7d876cecef66 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:18.698122 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0110f5f-d313-4cb9-aaf5-7d876cecef66 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:18.698398 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0110f5f-d313-4cb9-aaf5-7d876cecef66 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:18.699063 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0110f5f-d313-4cb9-aaf5-7d876cecef66 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:18.699362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0110f5f-d313-4cb9-aaf5-7d876cecef66 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:18.700746 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0110f5f-d313-4cb9-aaf5-7d876cecef66 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:18.706213 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 371/760] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 16 00:45:18 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:18.707056 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0110f5f-d313-4cb9-aaf5-7d876cecef66 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:18.726093 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3bb19a-1cfa-4f09-81f9-e19760277bf3 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:18.744562 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:18.764684 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c317287-53ee-4e10-85a0-c5cec42d29a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:18.765317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c317287-53ee-4e10-85a0-c5cec42d29a5 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:18.765599 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c317287-53ee-4e10-85a0-c5cec42d29a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:18.765880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c317287-53ee-4e10-85a0-c5cec42d29a5 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:18.766529 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c317287-53ee-4e10-85a0-c5cec42d29a5 None admin] RBAC: Adding query filter params (type=image) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:18.766840 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c317287-53ee-4e10-85a0-c5cec42d29a5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:18.768596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c317287-53ee-4e10-85a0-c5cec42d29a5 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:18.778269 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 390/761] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:18 2022] GET /identity/v3/services?type=image => generated 350 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:18.778938 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c317287-53ee-4e10-85a0-c5cec42d29a5 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:18.823786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0110f5f-d313-4cb9-aaf5-7d876cecef66 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:18.860494 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80350347-668b-4866-9ef5-4707b3d7e1ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:18.861137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80350347-668b-4866-9ef5-4707b3d7e1ed None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:18.861420 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80350347-668b-4866-9ef5-4707b3d7e1ed None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:18.861693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80350347-668b-4866-9ef5-4707b3d7e1ed None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:18.862286 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80350347-668b-4866-9ef5-4707b3d7e1ed None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:18.863716 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80350347-668b-4866-9ef5-4707b3d7e1ed None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:18.873262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-80350347-668b-4866-9ef5-4707b3d7e1ed None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 16 00:45:18.873262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:18.873262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:45:18.873262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:45:18.873262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:45:18.873262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:45:18.873262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:45:18.873262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:18.873262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 16 00:45:18.873262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:18.873262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:18.873262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:18.873262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:18.873262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:18.873262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:18.873262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:18.873262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:18.875363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:18.875363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:18.875363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:18.875363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:18.875363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:18.875363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:18.875363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:45:18.875363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:45:18.875363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:45:18.875363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:45:18.875363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:18.875363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:18.875363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:18.875363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:18.875363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:18.875363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:18.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:18.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:18.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:18.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:18.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:18.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:18.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:18.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:18.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:18.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:18.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:45:18.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:18.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 16 00:45:18.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:18.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 372/762] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 16 00:45:18 2022] GET /identity/v3/services/object-store => generated 93 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:18.878960 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80350347-668b-4866-9ef5-4707b3d7e1ed None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:18.880233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:18.899540 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c317287-53ee-4e10-85a0-c5cec42d29a5 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:18.940447 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33e94ce-9039-4ced-bf1e-7cda7e7e4d06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:18.941178 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33e94ce-9039-4ced-bf1e-7cda7e7e4d06 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:18.941596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33e94ce-9039-4ced-bf1e-7cda7e7e4d06 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:18.941945 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33e94ce-9039-4ced-bf1e-7cda7e7e4d06 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:18.951971 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33e94ce-9039-4ced-bf1e-7cda7e7e4d06 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:18.953407 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33e94ce-9039-4ced-bf1e-7cda7e7e4d06 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:18.968662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-a33e94ce-9039-4ced-bf1e-7cda7e7e4d06 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 16 00:45:18.968662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:18.968662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:18.968662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:18.968662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:18.968662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:18.968662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:18.968662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:18.968662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:18.968662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:18.968662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:18.968662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:18.968662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:45:18.968662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:45:18.968662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:45:18.968662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:18.970711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:18.971902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:18.971902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:18.971902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:45:18.971902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:45:18.971902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:45:18.971902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:45:18.971902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:45:18.971902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:45:18.971902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 16 00:45:18.971902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:18.971902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 391/763] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:45:18 2022] GET /identity/v3/users/cinder => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:18.972610 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33e94ce-9039-4ced-bf1e-7cda7e7e4d06 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:19.008453 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80350347-668b-4866-9ef5-4707b3d7e1ed None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:19.030847 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:19.051920 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33db7dd4-429d-431a-9940-db7baf417a7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:19.052798 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33db7dd4-429d-431a-9940-db7baf417a7d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:19.053148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33db7dd4-429d-431a-9940-db7baf417a7d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:19.054018 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33db7dd4-429d-431a-9940-db7baf417a7d None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:19.054878 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33db7dd4-429d-431a-9940-db7baf417a7d None admin] RBAC: Adding query filter params (type=compute) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:19.055264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33db7dd4-429d-431a-9940-db7baf417a7d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:19.057132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33db7dd4-429d-431a-9940-db7baf417a7d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:19.063659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 373/764] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 16 00:45:18 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:19.064682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33db7dd4-429d-431a-9940-db7baf417a7d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:19.107334 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33e94ce-9039-4ced-bf1e-7cda7e7e4d06 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:19.143674 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:19.148777 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200a7745-e98c-4d18-ac46-ef1a5b8a63f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:19.149501 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200a7745-e98c-4d18-ac46-ef1a5b8a63f7 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:19.149914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200a7745-e98c-4d18-ac46-ef1a5b8a63f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:19.150335 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200a7745-e98c-4d18-ac46-ef1a5b8a63f7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:19.151286 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200a7745-e98c-4d18-ac46-ef1a5b8a63f7 None admin] RBAC: Adding query filter params (interface=public, service_id=9be767711b0a487fb3ba5fdfd61baaca, region_id=RegionOne) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:19.151710 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200a7745-e98c-4d18-ac46-ef1a5b8a63f7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:19.153225 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200a7745-e98c-4d18-ac46-ef1a5b8a63f7 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:19.160755 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 392/765] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Oct 16 00:45:18 2022] GET /identity/v3/endpoints?service_id=9be767711b0a487fb3ba5fdfd61baaca&interface=public®ion_id=RegionOne => generated 194 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:19.163089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200a7745-e98c-4d18-ac46-ef1a5b8a63f7 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:19.190326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33db7dd4-429d-431a-9940-db7baf417a7d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:19.229855 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a97cfb3-ef67-46e0-825c-6c6c148ee331 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:19.230505 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a97cfb3-ef67-46e0-825c-6c6c148ee331 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:19.230883 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a97cfb3-ef67-46e0-825c-6c6c148ee331 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:19.231237 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a97cfb3-ef67-46e0-825c-6c6c148ee331 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:19.231941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a97cfb3-ef67-46e0-825c-6c6c148ee331 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:19.232306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a97cfb3-ef67-46e0-825c-6c6c148ee331 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:19.233853 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a97cfb3-ef67-46e0-825c-6c6c148ee331 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:19.240038 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 374/766] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 16 00:45:19 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:19.242042 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a97cfb3-ef67-46e0-825c-6c6c148ee331 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:19.327106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200a7745-e98c-4d18-ac46-ef1a5b8a63f7 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:19.337831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:19.364516 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a97cfb3-ef67-46e0-825c-6c6c148ee331 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:19.374260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274332aa-2f02-460b-b69b-614e77dedf00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:19.375270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274332aa-2f02-460b-b69b-614e77dedf00 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:19.375712 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274332aa-2f02-460b-b69b-614e77dedf00 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:19.376158 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274332aa-2f02-460b-b69b-614e77dedf00 None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:19.377353 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274332aa-2f02-460b-b69b-614e77dedf00 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:19.377785 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274332aa-2f02-460b-b69b-614e77dedf00 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:19.380386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274332aa-2f02-460b-b69b-614e77dedf00 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:19.407996 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:19.417751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2afac5c-a848-4188-8a2b-8a45e38c8f78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:19.418687 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2afac5c-a848-4188-8a2b-8a45e38c8f78 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:19.419172 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2afac5c-a848-4188-8a2b-8a45e38c8f78 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:19.419618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2afac5c-a848-4188-8a2b-8a45e38c8f78 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:19.420804 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2afac5c-a848-4188-8a2b-8a45e38c8f78 None admin] RBAC: Adding query filter params (interface=public, service_id=5ce2192c8fde4ec1b48e5031e898446c, region_id=RegionOne) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:19.421313 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2afac5c-a848-4188-8a2b-8a45e38c8f78 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:19.423841 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2afac5c-a848-4188-8a2b-8a45e38c8f78 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:19.433028 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 375/767] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Oct 16 00:45:19 2022] GET /identity/v3/endpoints?service_id=5ce2192c8fde4ec1b48e5031e898446c&interface=public®ion_id=RegionOne => generated 194 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:19.434123 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2afac5c-a848-4188-8a2b-8a45e38c8f78 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:19.435244 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 393/768] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 16 00:45:19 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:19.436249 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274332aa-2f02-460b-b69b-614e77dedf00 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:19.599159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274332aa-2f02-460b-b69b-614e77dedf00 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:19.620715 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2afac5c-a848-4188-8a2b-8a45e38c8f78 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:19.671601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f53ce1-6835-4700-959a-b51bb143fbee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:19.672680 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f53ce1-6835-4700-959a-b51bb143fbee None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:19.673186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f53ce1-6835-4700-959a-b51bb143fbee None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:19.673673 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f53ce1-6835-4700-959a-b51bb143fbee None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:19.674902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f53ce1-6835-4700-959a-b51bb143fbee None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:19.675390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f53ce1-6835-4700-959a-b51bb143fbee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:19.677853 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f53ce1-6835-4700-959a-b51bb143fbee None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:19.678628 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fccc887b-10fe-4457-a167-791db59f8ab8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:19.679406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccc887b-10fe-4457-a167-791db59f8ab8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:19.679819 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccc887b-10fe-4457-a167-791db59f8ab8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:19.680229 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccc887b-10fe-4457-a167-791db59f8ab8 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:19.681115 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fccc887b-10fe-4457-a167-791db59f8ab8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:19.683590 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fccc887b-10fe-4457-a167-791db59f8ab8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:19.691140 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 394/769] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 16 00:45:19 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:19.691140 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f53ce1-6835-4700-959a-b51bb143fbee None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:19.697859 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 376/770] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 16 00:45:19 2022] GET /identity/v3/services => generated 1957 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:45:19.699007 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fccc887b-10fe-4457-a167-791db59f8ab8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:19.746307 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:19.794482 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:19.866584 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fccc887b-10fe-4457-a167-791db59f8ab8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:19.915987 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7020b01-d1eb-4d25-b2ab-77d01abfbc71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:19.917146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7020b01-d1eb-4d25-b2ab-77d01abfbc71 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:19.917797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7020b01-d1eb-4d25-b2ab-77d01abfbc71 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:19.918403 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7020b01-d1eb-4d25-b2ab-77d01abfbc71 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:19.920633 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7020b01-d1eb-4d25-b2ab-77d01abfbc71 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:19.922567 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7020b01-d1eb-4d25-b2ab-77d01abfbc71 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:19.922770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f53ce1-6835-4700-959a-b51bb143fbee None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:19.959213 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 377/771] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:45:19 2022] POST /identity/v3/endpoints => generated 325 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:45:19.989138 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bc0698-7cf9-4634-968f-3c8c8562c89b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:19.990216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bc0698-7cf9-4634-968f-3c8c8562c89b None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:19.990647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bc0698-7cf9-4634-968f-3c8c8562c89b None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:19.991701 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bc0698-7cf9-4634-968f-3c8c8562c89b None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:19.992953 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bc0698-7cf9-4634-968f-3c8c8562c89b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:19.995376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bc0698-7cf9-4634-968f-3c8c8562c89b None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:20.008257 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 395/772] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 16 00:45:19 2022] GET /identity/v3/services => generated 1957 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:45:22.001017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7ed9a8-c63e-4c48-b267-9fa62c47365f None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:22.001555 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7ed9a8-c63e-4c48-b267-9fa62c47365f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:22.001972 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7ed9a8-c63e-4c48-b267-9fa62c47365f None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:22.004679 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 378/773] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:21 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:22.011236 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365a338a-a277-4de1-a04d-f3085cede090 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:22.011988 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365a338a-a277-4de1-a04d-f3085cede090 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:22.011988 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365a338a-a277-4de1-a04d-f3085cede090 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:22.031784 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-365a338a-a277-4de1-a04d-f3085cede090 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:22.033381 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-365a338a-a277-4de1-a04d-f3085cede090 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:22.113974 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 396/774] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:22 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:45:22.357555 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137a8e97-efe4-434e-aa61-cc9bdd86e58c None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:22.357845 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137a8e97-efe4-434e-aa61-cc9bdd86e58c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:22.358254 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137a8e97-efe4-434e-aa61-cc9bdd86e58c None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:22.359711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 379/775] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:45:22.366287 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66097ce4-d9e7-4ebd-b2b0-34d0deb063ea None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:22.366894 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66097ce4-d9e7-4ebd-b2b0-34d0deb063ea None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:22.367388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66097ce4-d9e7-4ebd-b2b0-34d0deb063ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:22.383161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137a8e97-efe4-434e-aa61-cc9bdd86e58c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:22.397896 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-66097ce4-d9e7-4ebd-b2b0-34d0deb063ea None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:22.401442 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-66097ce4-d9e7-4ebd-b2b0-34d0deb063ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:22.427135 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:22.503514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137a8e97-efe4-434e-aa61-cc9bdd86e58c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:22.512745 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 397/776] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:22 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:45:22.523666 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:22.546340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fc83c6-c05b-4983-9671-93772ef4c193 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:22.547025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fc83c6-c05b-4983-9671-93772ef4c193 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:22.547308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fc83c6-c05b-4983-9671-93772ef4c193 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:22.547583 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fc83c6-c05b-4983-9671-93772ef4c193 None admin] PATH_INFO: `/v3/services/image` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:22.548179 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fc83c6-c05b-4983-9671-93772ef4c193 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:22.549597 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fc83c6-c05b-4983-9671-93772ef4c193 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:22.559545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-48fc83c6-c05b-4983-9671-93772ef4c193 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 16 00:45:22.559545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:22.559545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:45:22.559545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:45:22.559545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:45:22.559545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:45:22.559545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:45:22.559545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:22.559545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 16 00:45:22.559545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:22.559545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:22.559545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:22.559545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:22.559545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:22.559545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:22.559545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:22.559545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:22.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:22.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:22.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:22.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:22.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:22.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:22.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:45:22.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:45:22.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:45:22.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:45:22.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:22.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:22.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:22.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:22.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:22.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:22.563002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:22.563002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:22.563002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:22.563002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:22.563002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:22.563002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:22.563002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:22.563002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:22.563002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:22.563002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:22.563002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:45:22.563002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:22.563002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 16 00:45:22.563002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:22.565753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 380/777] 10.222.0.25 () {56 vars in 1105 bytes} [Sun Oct 16 00:45:22 2022] GET /identity/v3/services/image => generated 86 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:22.566788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66097ce4-d9e7-4ebd-b2b0-34d0deb063ea None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:22.744478 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66097ce4-d9e7-4ebd-b2b0-34d0deb063ea None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:22.784082 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f806903-47ca-4449-9dd7-00f2ce389dac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:22.784708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f806903-47ca-4449-9dd7-00f2ce389dac None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:22.784984 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f806903-47ca-4449-9dd7-00f2ce389dac None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:22.785257 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f806903-47ca-4449-9dd7-00f2ce389dac None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:22.785889 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f806903-47ca-4449-9dd7-00f2ce389dac None admin] RBAC: Adding query filter params (name=image) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:22.786189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f806903-47ca-4449-9dd7-00f2ce389dac None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:22.787680 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f806903-47ca-4449-9dd7-00f2ce389dac None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:22.795386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 398/778] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:22 2022] GET /identity/v3/services?name=image => generated 122 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:22.799354 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f806903-47ca-4449-9dd7-00f2ce389dac None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:22.799919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fc83c6-c05b-4983-9671-93772ef4c193 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:22.818933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:22.918567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f806903-47ca-4449-9dd7-00f2ce389dac None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:22.980225 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f45341f-81fe-473e-ae75-0d2d8dc7089a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:22.980951 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f45341f-81fe-473e-ae75-0d2d8dc7089a None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:22.981173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f45341f-81fe-473e-ae75-0d2d8dc7089a None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:22.981444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f45341f-81fe-473e-ae75-0d2d8dc7089a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:22.982691 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f45341f-81fe-473e-ae75-0d2d8dc7089a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:22.984215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f45341f-81fe-473e-ae75-0d2d8dc7089a None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:22.986074 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 399/779] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:22 2022] GET /identity/v3/domains/default => generated 199 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:22.991322 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fc83c6-c05b-4983-9671-93772ef4c193 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:22.992809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f45341f-81fe-473e-ae75-0d2d8dc7089a None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:23.056757 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85ccce2-3a66-47e5-88ee-9daf58a673a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:23.057819 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85ccce2-3a66-47e5-88ee-9daf58a673a1 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:23.058339 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85ccce2-3a66-47e5-88ee-9daf58a673a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:23.058843 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85ccce2-3a66-47e5-88ee-9daf58a673a1 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:23.060036 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85ccce2-3a66-47e5-88ee-9daf58a673a1 None admin] RBAC: Adding query filter params (type=image) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:23.060608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85ccce2-3a66-47e5-88ee-9daf58a673a1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:23.063280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85ccce2-3a66-47e5-88ee-9daf58a673a1 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:23.073884 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 381/780] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:22 2022] GET /identity/v3/services?type=image => generated 350 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:23.080263 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85ccce2-3a66-47e5-88ee-9daf58a673a1 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:23.090486 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:23.119347 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f45341f-81fe-473e-ae75-0d2d8dc7089a None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:23.143400 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:23.160937 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7848a7d0-f394-47f0-8138-9f3f6cc05555 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:23.161809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7848a7d0-f394-47f0-8138-9f3f6cc05555 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:23.162245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7848a7d0-f394-47f0-8138-9f3f6cc05555 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:23.162738 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7848a7d0-f394-47f0-8138-9f3f6cc05555 None admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:23.163843 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7848a7d0-f394-47f0-8138-9f3f6cc05555 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:23.165292 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7848a7d0-f394-47f0-8138-9f3f6cc05555 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:23.203769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 400/781] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 16 00:45:22 2022] POST /identity/v3/projects => generated 311 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:45:23.203769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63923e2b-97f0-4aea-9658-a0c1db44cab5 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:23.203769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63923e2b-97f0-4aea-9658-a0c1db44cab5 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:23.203769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63923e2b-97f0-4aea-9658-a0c1db44cab5 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:23.203769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 401/782] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:23.208704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3a464f-3d36-4c5a-affc-433383084b72 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:23.209156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3a464f-3d36-4c5a-affc-433383084b72 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:23.209715 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3a464f-3d36-4c5a-affc-433383084b72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:23.233482 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85ccce2-3a66-47e5-88ee-9daf58a673a1 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:23.247706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-fc3a464f-3d36-4c5a-affc-433383084b72 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:23.250826 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-fc3a464f-3d36-4c5a-affc-433383084b72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:23.283024 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7a06d5-9e9a-4a6c-ab33-c79d05b47ee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:23.283698 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7a06d5-9e9a-4a6c-ab33-c79d05b47ee9 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:23.283980 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7a06d5-9e9a-4a6c-ab33-c79d05b47ee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:23.284258 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7a06d5-9e9a-4a6c-ab33-c79d05b47ee9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:23.284996 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7a06d5-9e9a-4a6c-ab33-c79d05b47ee9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:23.286593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7a06d5-9e9a-4a6c-ab33-c79d05b47ee9 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:23.314656 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 382/783] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:45:23 2022] POST /identity/v3/endpoints => generated 327 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:45:23.413787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 402/784] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:23 2022] POST /identity/v3/auth/tokens => generated 1962 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:45:23.689926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7a06d5-9e9a-4a6c-ab33-c79d05b47ee9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:23.804335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7a06d5-9e9a-4a6c-ab33-c79d05b47ee9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:23.806228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:23.856734 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc7a1d6-c67a-41a1-8ccd-09181d840135 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:23.858511 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc7a1d6-c67a-41a1-8ccd-09181d840135 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:23.859253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc7a1d6-c67a-41a1-8ccd-09181d840135 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:23.859990 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc7a1d6-c67a-41a1-8ccd-09181d840135 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:23.861441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc7a1d6-c67a-41a1-8ccd-09181d840135 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:23.864394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc7a1d6-c67a-41a1-8ccd-09181d840135 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:23.883659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-ccc7a1d6-c67a-41a1-8ccd-09181d840135 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 16 00:45:23.883659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:23.883659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:45:23.883659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:45:23.883659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:45:23.883659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:45:23.883659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:45:23.883659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:23.883659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 16 00:45:23.883659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:23.883659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:23.883659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:23.883659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:23.883659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:23.883659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:23.883659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:23.883659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:23.885751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:23.885751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:23.885751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:23.885751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:23.885751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:23.885751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:23.885751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:45:23.885751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:45:23.885751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:45:23.885751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:45:23.885751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:23.885751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:23.885751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:23.885751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:23.885751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:23.885751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:23.888686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:23.888686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:23.888686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:23.888686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:23.888686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:23.888686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:23.888686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:23.888686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:23.888686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:23.888686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:23.888686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:45:23.888686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:45:23.888686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 16 00:45:23.888686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:23.888686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 383/785] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:45:23 2022] GET /identity/v3/services/compute => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:23.892971 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3a464f-3d36-4c5a-affc-433383084b72 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:23.975901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:24.048095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3a464f-3d36-4c5a-affc-433383084b72 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:24.093829 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a54be37-d599-41ed-bed2-cc1a22ff073e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:24.094846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a54be37-d599-41ed-bed2-cc1a22ff073e None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:24.094846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a54be37-d599-41ed-bed2-cc1a22ff073e None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:24.095360 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a54be37-d599-41ed-bed2-cc1a22ff073e None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:24.095694 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a54be37-d599-41ed-bed2-cc1a22ff073e None admin] RBAC: Adding query filter params (name=compute) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:24.096004 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a54be37-d599-41ed-bed2-cc1a22ff073e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:24.097567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a54be37-d599-41ed-bed2-cc1a22ff073e None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:24.109946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 403/786] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 16 00:45:23 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:24.112055 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc7a1d6-c67a-41a1-8ccd-09181d840135 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:24.251731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc7a1d6-c67a-41a1-8ccd-09181d840135 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:24.320645 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b52f14-a74d-4a12-9655-d5b7931b5033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:24.321727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b52f14-a74d-4a12-9655-d5b7931b5033 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:24.322234 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b52f14-a74d-4a12-9655-d5b7931b5033 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:24.322584 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b52f14-a74d-4a12-9655-d5b7931b5033 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:24.323609 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b52f14-a74d-4a12-9655-d5b7931b5033 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:24.324174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b52f14-a74d-4a12-9655-d5b7931b5033 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:24.326199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b52f14-a74d-4a12-9655-d5b7931b5033 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:24.341942 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 384/787] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 16 00:45:24 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:24.344498 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a54be37-d599-41ed-bed2-cc1a22ff073e None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:24.499106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a54be37-d599-41ed-bed2-cc1a22ff073e None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:24.530411 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:24.549452 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c92634-34f9-4862-b00a-005092993840 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:24.550546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c92634-34f9-4862-b00a-005092993840 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:24.551190 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c92634-34f9-4862-b00a-005092993840 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:24.551851 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c92634-34f9-4862-b00a-005092993840 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:24.554389 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c92634-34f9-4862-b00a-005092993840 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:24.557139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c92634-34f9-4862-b00a-005092993840 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:24.593649 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 404/788] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:45:24 2022] POST /identity/v3/endpoints => generated 334 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:45:25.833705 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f005d40-5b40-4a31-aa1e-29b20eb2abaf None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:25.834844 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f005d40-5b40-4a31-aa1e-29b20eb2abaf None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:25.835181 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f005d40-5b40-4a31-aa1e-29b20eb2abaf None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:25.836891 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 385/789] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:25.842854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8a56ef-10ba-4029-b550-e795e8ef2c83 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:25.843460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8a56ef-10ba-4029-b550-e795e8ef2c83 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:25.843460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8a56ef-10ba-4029-b550-e795e8ef2c83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:25.862437 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-0a8a56ef-10ba-4029-b550-e795e8ef2c83 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:25.863892 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-0a8a56ef-10ba-4029-b550-e795e8ef2c83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:25.872576 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47141860-08b0-4344-84dc-36df0b5f8c96 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:25.872900 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47141860-08b0-4344-84dc-36df0b5f8c96 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:25.873791 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47141860-08b0-4344-84dc-36df0b5f8c96 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:25.874636 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 386/790] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:45:25.879928 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c36536b-99f5-4049-aa7e-90010ce9f7de None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:25.880215 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c36536b-99f5-4049-aa7e-90010ce9f7de None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:25.880546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c36536b-99f5-4049-aa7e-90010ce9f7de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:25.900294 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-7c36536b-99f5-4049-aa7e-90010ce9f7de None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:25.901527 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7c36536b-99f5-4049-aa7e-90010ce9f7de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:25.932913 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:25.960867 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 405/791] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:25 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:25.993746 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 387/792] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:25 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:45:26.235569 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8a56ef-10ba-4029-b550-e795e8ef2c83 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:26.277187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c36536b-99f5-4049-aa7e-90010ce9f7de None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:26.384444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8a56ef-10ba-4029-b550-e795e8ef2c83 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:26.387735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:26.411488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c36536b-99f5-4049-aa7e-90010ce9f7de None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:26.439352 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19676137-7c43-47ca-95f3-c64f30e5b0e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:26.439985 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19676137-7c43-47ca-95f3-c64f30e5b0e1 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:26.440263 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19676137-7c43-47ca-95f3-c64f30e5b0e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:26.440533 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19676137-7c43-47ca-95f3-c64f30e5b0e1 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:26.441216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19676137-7c43-47ca-95f3-c64f30e5b0e1 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:26.442604 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19676137-7c43-47ca-95f3-c64f30e5b0e1 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:26.451895 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.sql.core [None req-19676137-7c43-47ca-95f3-c64f30e5b0e1 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 16 00:45:26.451895 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 16 00:45:26.451895 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [parameters: {'id': '67da488b3db04d5992caa5254b7bfbdc', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 16 00:45:26.451895 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81486) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 16 00:45:26.460003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-19676137-7c43-47ca-95f3-c64f30e5b0e1 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 16 00:45:26.460003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:26.460003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 16 00:45:26.460003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 16 00:45:26.460003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 16 00:45:26.460003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 16 00:45:26.460003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 16 00:45:26.460003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application result = self._query(query) Oct 16 00:45:26.460003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 16 00:45:26.460003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application conn.query(q) Oct 16 00:45:26.460003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 16 00:45:26.460003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 16 00:45:26.460003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 16 00:45:26.460003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application result.read() Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 16 00:45:26.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 16 00:45:26.464462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application next(self.gen) Oct 16 00:45:26.464462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 16 00:45:26.464462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application yield resource Oct 16 00:45:26.464462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 16 00:45:26.464462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application next(self.gen) Oct 16 00:45:26.464462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 16 00:45:26.464462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self.session.rollback() Oct 16 00:45:26.464462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 16 00:45:26.464462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self.force_reraise() Oct 16 00:45:26.464462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 16 00:45:26.464462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise self.value Oct 16 00:45:26.464462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 16 00:45:26.464462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 16 00:45:26.464462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 16 00:45:26.464462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application session.commit() Oct 16 00:45:26.464462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 16 00:45:26.464462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 16 00:45:26.465561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 16 00:45:26.465561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._prepare_impl() Oct 16 00:45:26.465561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 16 00:45:26.465561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self.session.flush() Oct 16 00:45:26.465561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 16 00:45:26.465561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._flush(objects) Oct 16 00:45:26.465561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 16 00:45:26.465561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 16 00:45:26.465561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 16 00:45:26.465561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application compat.raise_( Oct 16 00:45:26.465561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 16 00:45:26.465561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception Oct 16 00:45:26.465561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 16 00:45:26.465561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application flush_context.execute() Oct 16 00:45:26.465561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 16 00:45:26.465561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rec.execute(self) Oct 16 00:45:26.465561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 16 00:45:26.466747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 16 00:45:26.466747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 16 00:45:26.466747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 16 00:45:26.466747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 16 00:45:26.466747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 16 00:45:26.466747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 16 00:45:26.466747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 16 00:45:26.466747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 16 00:45:26.466747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 16 00:45:26.466747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 16 00:45:26.466747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 16 00:45:26.466747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 16 00:45:26.466747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 16 00:45:26.466747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 16 00:45:26.466747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 16 00:45:26.467863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 16 00:45:26.467863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception Oct 16 00:45:26.467863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 16 00:45:26.467863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 16 00:45:26.467863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 16 00:45:26.467863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 16 00:45:26.467863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 16 00:45:26.467863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application result = self._query(query) Oct 16 00:45:26.467863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 16 00:45:26.467863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application conn.query(q) Oct 16 00:45:26.467863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 16 00:45:26.467863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 16 00:45:26.467863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 16 00:45:26.467863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application result.read() Oct 16 00:45:26.467863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 16 00:45:26.467863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 16 00:45:26.467863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 16 00:45:26.469058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 16 00:45:26.469058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 16 00:45:26.469058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 16 00:45:26.469058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 16 00:45:26.469058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 16 00:45:26.469058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 16 00:45:26.469058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 16 00:45:26.469058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [parameters: {'id': '67da488b3db04d5992caa5254b7bfbdc', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 16 00:45:26.469058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 16 00:45:26.469058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:26.469058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:26.469058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:26.469058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:26.469058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 16 00:45:26.469058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 16 00:45:26.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 16 00:45:26.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 16 00:45:26.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 16 00:45:26.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:26.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:45:26.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:26.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:26.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:26.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:26.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:26.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:26.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:26.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:26.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:26.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:26.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:26.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:26.471653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 16 00:45:26.471653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 16 00:45:26.471653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:26.471653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:26.471653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 16 00:45:26.471653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 16 00:45:26.471653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:26.471653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:26.471653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 16 00:45:26.471653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 16 00:45:26.471653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 16 00:45:26.471653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:26.471653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 406/793] 10.222.0.25 () {60 vars in 1137 bytes} [Sun Oct 16 00:45:26 2022] POST /identity/v3/domains => generated 190 bytes in 227 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:45:26.473268 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19676137-7c43-47ca-95f3-c64f30e5b0e1 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:26.484534 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f236a8-b44b-4af5-9dd0-85431918168c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:26.485557 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f236a8-b44b-4af5-9dd0-85431918168c None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:26.486076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f236a8-b44b-4af5-9dd0-85431918168c None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:26.486513 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f236a8-b44b-4af5-9dd0-85431918168c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:26.488452 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f236a8-b44b-4af5-9dd0-85431918168c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:26.491181 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f236a8-b44b-4af5-9dd0-85431918168c None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:26.498917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 388/794] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:26 2022] GET /identity/v3/domains/default => generated 199 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:26.500613 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f236a8-b44b-4af5-9dd0-85431918168c None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:26.554107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:26.583400 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19676137-7c43-47ca-95f3-c64f30e5b0e1 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:26.619529 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202c2cba-f74f-495e-84e6-fdc79cc754b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:26.620191 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202c2cba-f74f-495e-84e6-fdc79cc754b1 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:26.620482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202c2cba-f74f-495e-84e6-fdc79cc754b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:26.620738 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202c2cba-f74f-495e-84e6-fdc79cc754b1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:26.631603 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202c2cba-f74f-495e-84e6-fdc79cc754b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:26.633084 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202c2cba-f74f-495e-84e6-fdc79cc754b1 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:26.652312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-202c2cba-f74f-495e-84e6-fdc79cc754b1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:45:26.652312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:26.652312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:26.652312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:26.652312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:26.652312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:26.652312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:26.652312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:26.652312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:26.652312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:26.652312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:26.652312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:26.652312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:45:26.652312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:45:26.652312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:45:26.652312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:45:26.653343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:26.653343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:26.653343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:26.653343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:26.653343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:26.653343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:26.653343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:26.653343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:26.653343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:26.653343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:26.653343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:26.653343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:26.653343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:26.653343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:26.653343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:26.653343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:26.653343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:45:26.654246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:45:26.654246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:45:26.654246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:26.654246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 407/795] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:26.659479 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202c2cba-f74f-495e-84e6-fdc79cc754b1 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:26.684476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:26.686706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f236a8-b44b-4af5-9dd0-85431918168c None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:26.740180 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869f91d6-2697-410b-87ca-df108959852f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:26.741783 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869f91d6-2697-410b-87ca-df108959852f None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:26.742048 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869f91d6-2697-410b-87ca-df108959852f None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:26.742386 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869f91d6-2697-410b-87ca-df108959852f None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:26.743064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869f91d6-2697-410b-87ca-df108959852f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:26.744508 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869f91d6-2697-410b-87ca-df108959852f None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:26.775301 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 389/796] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:45:26 2022] POST /identity/v3/users => generated 285 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:45:26.816767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202c2cba-f74f-495e-84e6-fdc79cc754b1 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:26.864712 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a781620d-9e01-416e-b296-a167c2cb7ec1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:26.866173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a781620d-9e01-416e-b296-a167c2cb7ec1 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:26.866462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a781620d-9e01-416e-b296-a167c2cb7ec1 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:26.867127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a781620d-9e01-416e-b296-a167c2cb7ec1 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:26.867910 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a781620d-9e01-416e-b296-a167c2cb7ec1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:26.868284 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a781620d-9e01-416e-b296-a167c2cb7ec1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:26.869869 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a781620d-9e01-416e-b296-a167c2cb7ec1 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:26.890685 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 408/797] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:45:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:27.075069 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6689f0f5-66f5-4852-af68-b4666179cbc2 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:27.075914 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6689f0f5-66f5-4852-af68-b4666179cbc2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:27.076079 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6689f0f5-66f5-4852-af68-b4666179cbc2 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:27.077665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 390/798] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:27 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:27.086200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dde5b69-b658-40e3-bdd7-526f621a4ff9 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:27.086640 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dde5b69-b658-40e3-bdd7-526f621a4ff9 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:27.087046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dde5b69-b658-40e3-bdd7-526f621a4ff9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:27.108290 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-8dde5b69-b658-40e3-bdd7-526f621a4ff9 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:27.109636 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8dde5b69-b658-40e3-bdd7-526f621a4ff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:27.204461 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 409/799] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:27.480029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6689f0f5-66f5-4852-af68-b4666179cbc2 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:27.575986 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:27.596908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6689f0f5-66f5-4852-af68-b4666179cbc2 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:27.649031 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca567ec0-c16a-4654-a860-83e0026e929e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:27.649489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca567ec0-c16a-4654-a860-83e0026e929e None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:27.649766 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca567ec0-c16a-4654-a860-83e0026e929e None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:27.650061 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca567ec0-c16a-4654-a860-83e0026e929e None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:27.663077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca567ec0-c16a-4654-a860-83e0026e929e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:45:27.672912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-ca567ec0-c16a-4654-a860-83e0026e929e None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 16 00:45:27.672912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:27.672912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:27.672912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:27.672912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:27.672912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:27.672912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:27.672912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:27.672912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:27.672912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:27.672912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:27.672912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:27.672912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:45:27.672912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:45:27.672912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:45:27.672912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:45:27.679033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:45:27.679033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:45:27.679033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:45:27.679033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:45:27.679033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:27.679033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:27.679033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:27.679033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:27.679033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:27.679033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:27.679033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:27.679033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:27.679033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:27.679033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:27.679033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:27.679033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:27.680556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:27.680556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:27.680556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:27.680556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:27.680556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:45:27.680556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:45:27.680556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:45:27.680556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:45:27.680556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:45:27.680556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:45:27.680556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 16 00:45:27.680556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:27.680556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 391/800] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Oct 16 00:45:27 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:27.681754 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dde5b69-b658-40e3-bdd7-526f621a4ff9 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:27.693332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:27.791463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dde5b69-b658-40e3-bdd7-526f621a4ff9 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:27.830818 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1a802f-10f9-41d7-a8a2-02c95fb5ad87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:27.831471 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1a802f-10f9-41d7-a8a2-02c95fb5ad87 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:27.831750 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1a802f-10f9-41d7-a8a2-02c95fb5ad87 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:27.832023 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1a802f-10f9-41d7-a8a2-02c95fb5ad87 None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:27.832683 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1a802f-10f9-41d7-a8a2-02c95fb5ad87 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:27.832986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1a802f-10f9-41d7-a8a2-02c95fb5ad87 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:27.834430 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1a802f-10f9-41d7-a8a2-02c95fb5ad87 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:27.843866 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 410/801] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Oct 16 00:45:27 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:27.850759 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca567ec0-c16a-4654-a860-83e0026e929e None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:27.968147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca567ec0-c16a-4654-a860-83e0026e929e None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:28.008323 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24a339f-54b2-42f4-9155-cb7469d0ec97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:28.008984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24a339f-54b2-42f4-9155-cb7469d0ec97 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:28.009268 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24a339f-54b2-42f4-9155-cb7469d0ec97 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:28.009544 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24a339f-54b2-42f4-9155-cb7469d0ec97 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:28.022630 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24a339f-54b2-42f4-9155-cb7469d0ec97 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:28.024236 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24a339f-54b2-42f4-9155-cb7469d0ec97 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:28.027390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:28.041615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-d24a339f-54b2-42f4-9155-cb7469d0ec97 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:45:28.041615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:28.041615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:28.041615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:28.041615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:28.041615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:28.041615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:28.041615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:28.041615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:28.041615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:28.041615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:28.041615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:28.041615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:45:28.041615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:45:28.041615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:45:28.041615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:45:28.042749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:28.042749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:28.042749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:28.042749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:28.042749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:28.042749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:28.042749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:28.042749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:28.042749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:28.042749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:28.042749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:28.042749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:28.042749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:28.042749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:28.042749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:28.042749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:28.042749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:45:28.044399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:45:28.044399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:45:28.044399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:28.044399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 392/802] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:28.049571 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1a802f-10f9-41d7-a8a2-02c95fb5ad87 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:28.168151 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:28.204317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1a802f-10f9-41d7-a8a2-02c95fb5ad87 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:28.242951 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383bde57-2fb4-47eb-aa72-c42a108a0c25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:28.243488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383bde57-2fb4-47eb-aa72-c42a108a0c25 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:28.243771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383bde57-2fb4-47eb-aa72-c42a108a0c25 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:28.244047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383bde57-2fb4-47eb-aa72-c42a108a0c25 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:28.244704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383bde57-2fb4-47eb-aa72-c42a108a0c25 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:28.245012 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383bde57-2fb4-47eb-aa72-c42a108a0c25 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:28.246407 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383bde57-2fb4-47eb-aa72-c42a108a0c25 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:28.260592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 411/803] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:45:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:28.266498 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24a339f-54b2-42f4-9155-cb7469d0ec97 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:28.378944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24a339f-54b2-42f4-9155-cb7469d0ec97 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:28.416093 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e55ffe3-bee9-47a2-bcd2-f86ff048649a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:28.416735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e55ffe3-bee9-47a2-bcd2-f86ff048649a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:28.417013 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e55ffe3-bee9-47a2-bcd2-f86ff048649a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:28.417288 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e55ffe3-bee9-47a2-bcd2-f86ff048649a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:28.425637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e55ffe3-bee9-47a2-bcd2-f86ff048649a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:28.427179 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e55ffe3-bee9-47a2-bcd2-f86ff048649a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:28.439428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-7e55ffe3-bee9-47a2-bcd2-f86ff048649a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 16 00:45:28.439428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:28.439428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:28.439428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:28.439428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:28.439428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:28.439428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:28.439428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:28.439428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:28.439428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:28.439428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:28.439428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:28.439428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:45:28.439428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:45:28.439428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:45:28.439428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:28.441457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:28.444044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:28.444044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:28.444044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:45:28.444044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:45:28.444044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:45:28.444044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:45:28.444044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:45:28.444044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:45:28.444044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 16 00:45:28.444044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:28.444044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 393/804] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Oct 16 00:45:28 2022] GET /identity/v3/users/nova => generated 82 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:28.447172 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383bde57-2fb4-47eb-aa72-c42a108a0c25 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:28.567742 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383bde57-2fb4-47eb-aa72-c42a108a0c25 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:28.601450 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:28.607674 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8de4cb-31eb-4653-9d2a-2ef8f11b8a48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:28.608312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8de4cb-31eb-4653-9d2a-2ef8f11b8a48 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:28.608592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8de4cb-31eb-4653-9d2a-2ef8f11b8a48 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:28.608864 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8de4cb-31eb-4653-9d2a-2ef8f11b8a48 None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:28.609598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8de4cb-31eb-4653-9d2a-2ef8f11b8a48 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:28.609909 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8de4cb-31eb-4653-9d2a-2ef8f11b8a48 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:28.611422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8de4cb-31eb-4653-9d2a-2ef8f11b8a48 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:28.633178 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 412/805] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Oct 16 00:45:28 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:28.639182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e55ffe3-bee9-47a2-bcd2-f86ff048649a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:28.702900 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:28.753514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e55ffe3-bee9-47a2-bcd2-f86ff048649a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:28.792301 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018f5811-3102-44e0-a7f6-0ceddbf1c753 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:28.792948 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018f5811-3102-44e0-a7f6-0ceddbf1c753 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:28.793230 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018f5811-3102-44e0-a7f6-0ceddbf1c753 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:28.793504 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018f5811-3102-44e0-a7f6-0ceddbf1c753 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:28.805847 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018f5811-3102-44e0-a7f6-0ceddbf1c753 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:28.807407 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018f5811-3102-44e0-a7f6-0ceddbf1c753 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:28.824224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-018f5811-3102-44e0-a7f6-0ceddbf1c753 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:45:28.824224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:28.824224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:28.824224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:28.824224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:28.824224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:28.824224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:28.824224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:28.824224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:28.824224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:28.824224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:28.824224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:28.824224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:45:28.824224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:45:28.824224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:45:28.824224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:45:28.825827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:28.825827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:28.825827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:28.825827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:28.825827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:28.825827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:28.825827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:28.825827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:28.825827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:28.825827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:28.825827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:28.825827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:28.825827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:28.825827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:28.825827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:28.825827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:28.825827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:45:28.827558 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:45:28.827558 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:45:28.827558 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:28.830706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 394/806] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:28.831864 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8de4cb-31eb-4653-9d2a-2ef8f11b8a48 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:28.948883 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8de4cb-31eb-4653-9d2a-2ef8f11b8a48 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:28.988357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccad8232-e44b-436b-bd5a-14a30c4f3776 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:28.988978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccad8232-e44b-436b-bd5a-14a30c4f3776 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:28.989255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccad8232-e44b-436b-bd5a-14a30c4f3776 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:28.989523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccad8232-e44b-436b-bd5a-14a30c4f3776 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:28.990162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccad8232-e44b-436b-bd5a-14a30c4f3776 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:28.991024 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccad8232-e44b-436b-bd5a-14a30c4f3776 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:28.992388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccad8232-e44b-436b-bd5a-14a30c4f3776 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:29.004226 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 413/807] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:45:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:29.010546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018f5811-3102-44e0-a7f6-0ceddbf1c753 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:29.131307 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018f5811-3102-44e0-a7f6-0ceddbf1c753 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:29.143563 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:29.175163 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d5f95c-0506-403e-a951-2b905e0cdc3d None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:29.175809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d5f95c-0506-403e-a951-2b905e0cdc3d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:29.175809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d5f95c-0506-403e-a951-2b905e0cdc3d None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:29.177087 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 414/808] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:45:29.183605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c80ea8-2995-4d1a-afee-acbc40260562 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:29.183933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c80ea8-2995-4d1a-afee-acbc40260562 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:29.184175 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c80ea8-2995-4d1a-afee-acbc40260562 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:29.184938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d68a52-c9b1-4ae5-b075-b16395409eb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:29.185734 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d68a52-c9b1-4ae5-b075-b16395409eb6 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:29.186155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d68a52-c9b1-4ae5-b075-b16395409eb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:29.186595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d68a52-c9b1-4ae5-b075-b16395409eb6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:29.197471 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d68a52-c9b1-4ae5-b075-b16395409eb6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:29.198998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d68a52-c9b1-4ae5-b075-b16395409eb6 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:29.208484 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-73c80ea8-2995-4d1a-afee-acbc40260562 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:29.210113 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-73c80ea8-2995-4d1a-afee-acbc40260562 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:29.212622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-16d68a52-c9b1-4ae5-b075-b16395409eb6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:45:29.212622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:29.212622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:29.212622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:29.212622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:29.212622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:29.212622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:29.212622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:29.212622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:29.212622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:29.212622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:29.212622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:29.212622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:45:29.212622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:45:29.212622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:45:29.213672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:45:29.213672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:29.213672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:29.213672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:29.213672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:29.213672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:29.213672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:29.213672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:29.213672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:29.213672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:29.213672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:29.213672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:29.213672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:29.213672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:29.213672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:29.213672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:29.213672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:29.214963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:45:29.214963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:45:29.214963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:45:29.214963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:45:29.214963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:45:29.214963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:45:29.214963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:45:29.214963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:29.226554 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 395/809] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:45:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:29.226554 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d68a52-c9b1-4ae5-b075-b16395409eb6 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:29.284678 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:29.317011 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 415/810] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:45:29.372803 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d68a52-c9b1-4ae5-b075-b16395409eb6 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:29.423395 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ef104f-534f-41f7-b318-809d83b25128 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:29.424128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ef104f-534f-41f7-b318-809d83b25128 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:29.424546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ef104f-534f-41f7-b318-809d83b25128 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:29.424957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ef104f-534f-41f7-b318-809d83b25128 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:29.425831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ef104f-534f-41f7-b318-809d83b25128 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:29.426266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ef104f-534f-41f7-b318-809d83b25128 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:29.428292 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ef104f-534f-41f7-b318-809d83b25128 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:29.442453 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 396/811] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:45:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:29.449712 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c80ea8-2995-4d1a-afee-acbc40260562 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:29.558619 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c80ea8-2995-4d1a-afee-acbc40260562 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:29.592614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ef104f-534f-41f7-b318-809d83b25128 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:29.597222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c37172b-0611-4675-8032-45fb050ec66e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:29.597911 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c37172b-0611-4675-8032-45fb050ec66e None admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:29.598190 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c37172b-0611-4675-8032-45fb050ec66e None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:29.598471 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c37172b-0611-4675-8032-45fb050ec66e None admin] PATH_INFO: `/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/4eba26813e154595a12a67fe4be5eeff/roles/7c38bf8daea94adcaaffae01c4253da9` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:29.601016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c37172b-0611-4675-8032-45fb050ec66e None admin] RBAC: Authorizing `identity:create_grant(project_id=7deb2c3dae214df7895b40e7339bd6ea, user_id=4eba26813e154595a12a67fe4be5eeff, role_id=7c38bf8daea94adcaaffae01c4253da9)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:29.602497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c37172b-0611-4675-8032-45fb050ec66e None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:29.617406 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 416/812] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:45:29 2022] PUT /identity/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/4eba26813e154595a12a67fe4be5eeff/roles/7c38bf8daea94adcaaffae01c4253da9 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:45:29.633374 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:29.724010 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ef104f-534f-41f7-b318-809d83b25128 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:29.765812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973b8047-f11e-401f-b8b8-51651a5a531d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:29.766496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973b8047-f11e-401f-b8b8-51651a5a531d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:29.766780 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973b8047-f11e-401f-b8b8-51651a5a531d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:29.767097 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973b8047-f11e-401f-b8b8-51651a5a531d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:29.781225 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973b8047-f11e-401f-b8b8-51651a5a531d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:45:29.786532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-973b8047-f11e-401f-b8b8-51651a5a531d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:45:29.786532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:29.786532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:29.786532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:29.786532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:29.786532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:29.786532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:29.786532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:29.786532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:29.786532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:29.786532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:29.786532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:29.786532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:45:29.786532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:45:29.786532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:45:29.786532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:45:29.788678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:45:29.788678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:45:29.788678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:45:29.788678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:45:29.788678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:29.788678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:29.788678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:29.788678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:29.788678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:29.788678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:29.788678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:29.788678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:29.788678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:29.788678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:29.788678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:29.788678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:29.790419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:29.790419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:29.790419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:29.790419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:29.790419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:45:29.790419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:45:29.790419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:45:29.790419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:45:29.790419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:45:29.790419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:45:29.790419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:45:29.790419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:29.790419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 397/813] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:45:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:45:29.794316 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c37172b-0611-4675-8032-45fb050ec66e None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:29.867497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:29.937811 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c37172b-0611-4675-8032-45fb050ec66e None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:29.975737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84ecf58-3b60-4951-b607-5d590dc8c04b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:29.976420 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84ecf58-3b60-4951-b607-5d590dc8c04b None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:29.976699 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84ecf58-3b60-4951-b607-5d590dc8c04b None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:29.976969 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84ecf58-3b60-4951-b607-5d590dc8c04b None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:29.977640 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84ecf58-3b60-4951-b607-5d590dc8c04b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:29.977942 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84ecf58-3b60-4951-b607-5d590dc8c04b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:29.979356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84ecf58-3b60-4951-b607-5d590dc8c04b None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:29.988219 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 417/814] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:45:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:29.994220 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973b8047-f11e-401f-b8b8-51651a5a531d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:30.095494 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:30.109793 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973b8047-f11e-401f-b8b8-51651a5a531d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:30.146832 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fd6a0c-ea50-4540-9cc7-309f064470d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:30.147508 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fd6a0c-ea50-4540-9cc7-309f064470d8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:30.147782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fd6a0c-ea50-4540-9cc7-309f064470d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:30.148052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fd6a0c-ea50-4540-9cc7-309f064470d8 None admin] PATH_INFO: `/v3/users/c5996cb0dc9843c6aa98fb07e2cd740c` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:30.164206 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fd6a0c-ea50-4540-9cc7-309f064470d8 None admin] RBAC: Authorizing `identity:get_user(user_id=c5996cb0dc9843c6aa98fb07e2cd740c)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:30.165762 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fd6a0c-ea50-4540-9cc7-309f064470d8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:30.167626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 398/815] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:45:29 2022] GET /identity/v3/users/c5996cb0dc9843c6aa98fb07e2cd740c => generated 285 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:30.174443 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84ecf58-3b60-4951-b607-5d590dc8c04b None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:30.297489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84ecf58-3b60-4951-b607-5d590dc8c04b None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:30.299206 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:30.351260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92fdefa-28f8-4eed-967c-63d8b84b633f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:30.351932 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92fdefa-28f8-4eed-967c-63d8b84b633f None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:30.352220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92fdefa-28f8-4eed-967c-63d8b84b633f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:30.352497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92fdefa-28f8-4eed-967c-63d8b84b633f None admin] PATH_INFO: `/v3/projects/d6bc095e9a044f6c8aa08c2ba3d79489` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:30.353674 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92fdefa-28f8-4eed-967c-63d8b84b633f None admin] RBAC: Authorizing `identity:get_project(project_id=d6bc095e9a044f6c8aa08c2ba3d79489)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:30.355161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92fdefa-28f8-4eed-967c-63d8b84b633f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:30.356834 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 418/816] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:45:30 2022] GET /identity/v3/projects/d6bc095e9a044f6c8aa08c2ba3d79489 => generated 311 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:30.363647 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fd6a0c-ea50-4540-9cc7-309f064470d8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:30.489705 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fd6a0c-ea50-4540-9cc7-309f064470d8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:30.528645 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1569d04d-1601-4236-b4b1-076410f163f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:30.529337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1569d04d-1601-4236-b4b1-076410f163f3 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:30.529622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1569d04d-1601-4236-b4b1-076410f163f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:30.529907 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1569d04d-1601-4236-b4b1-076410f163f3 None admin] PATH_INFO: `/v3/projects/d6bc095e9a044f6c8aa08c2ba3d79489/users/c5996cb0dc9843c6aa98fb07e2cd740c/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:30.532118 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1569d04d-1601-4236-b4b1-076410f163f3 None admin] RBAC: Authorizing `identity:create_grant(project_id=d6bc095e9a044f6c8aa08c2ba3d79489, user_id=c5996cb0dc9843c6aa98fb07e2cd740c, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:30.533617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1569d04d-1601-4236-b4b1-076410f163f3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:30.550884 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 399/817] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:45:30 2022] PUT /identity/v3/projects/d6bc095e9a044f6c8aa08c2ba3d79489/users/c5996cb0dc9843c6aa98fb07e2cd740c/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:45:31.852402 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ebf2b8-32f0-435f-8a8e-0138ea9b876f None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:31.853044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ebf2b8-32f0-435f-8a8e-0138ea9b876f None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:31.853256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ebf2b8-32f0-435f-8a8e-0138ea9b876f None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:31.855736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 419/818] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:31 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:31.861733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6e399f-e363-4b9c-8a21-5bf638f38b4e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:31.862501 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6e399f-e363-4b9c-8a21-5bf638f38b4e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:31.862668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6e399f-e363-4b9c-8a21-5bf638f38b4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:31.876578 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:31.884496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-cf6e399f-e363-4b9c-8a21-5bf638f38b4e None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:31.886171 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cf6e399f-e363-4b9c-8a21-5bf638f38b4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:31.963790 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 400/819] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:32.233758 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ebf2b8-32f0-435f-8a8e-0138ea9b876f None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:32.342803 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ebf2b8-32f0-435f-8a8e-0138ea9b876f None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:32.379176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf921811-a17f-478d-8f2f-02ce686812e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:32.379830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf921811-a17f-478d-8f2f-02ce686812e6 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:32.380104 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf921811-a17f-478d-8f2f-02ce686812e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:32.380372 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf921811-a17f-478d-8f2f-02ce686812e6 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:32.401464 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf921811-a17f-478d-8f2f-02ce686812e6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:45:32.404250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-cf921811-a17f-478d-8f2f-02ce686812e6 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 16 00:45:32.404250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:32.404250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:32.404250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:32.404250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:32.404250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:32.404250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:32.404250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:32.404250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:32.404250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:32.404250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:32.404250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:32.404250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:45:32.404250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:45:32.404250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:45:32.404250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:45:32.405473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:45:32.405473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:45:32.405473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:45:32.405473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:45:32.405473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:32.405473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:32.405473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:32.405473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:32.405473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:32.405473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:32.405473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:32.405473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:32.405473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:32.405473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:32.405473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:32.405473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:32.406422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:32.406422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:32.406422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:32.406422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:32.406422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:45:32.406422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:45:32.406422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:45:32.406422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:45:32.406422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:45:32.406422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:45:32.406422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 16 00:45:32.406422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:32.406422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 420/820] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Oct 16 00:45:32 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:32.412649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6e399f-e363-4b9c-8a21-5bf638f38b4e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:32.512035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6e399f-e363-4b9c-8a21-5bf638f38b4e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:32.546018 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aacc6a8-c787-40cc-8982-824abe2ec3ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:32.546631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aacc6a8-c787-40cc-8982-824abe2ec3ac None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:32.547008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aacc6a8-c787-40cc-8982-824abe2ec3ac None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:32.547289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aacc6a8-c787-40cc-8982-824abe2ec3ac None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:32.547997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aacc6a8-c787-40cc-8982-824abe2ec3ac None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:32.548305 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aacc6a8-c787-40cc-8982-824abe2ec3ac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:32.549682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aacc6a8-c787-40cc-8982-824abe2ec3ac None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:32.552051 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:32.559408 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 401/821] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Oct 16 00:45:32 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:32.565295 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf921811-a17f-478d-8f2f-02ce686812e6 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:32.597206 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:32.672303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf921811-a17f-478d-8f2f-02ce686812e6 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:32.706029 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14653283-d2dc-445f-98f1-19d322f3d1cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:32.706662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14653283-d2dc-445f-98f1-19d322f3d1cf None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:32.706979 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14653283-d2dc-445f-98f1-19d322f3d1cf None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:32.707251 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14653283-d2dc-445f-98f1-19d322f3d1cf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:32.718490 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14653283-d2dc-445f-98f1-19d322f3d1cf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:32.719971 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14653283-d2dc-445f-98f1-19d322f3d1cf None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:32.734034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-14653283-d2dc-445f-98f1-19d322f3d1cf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:45:32.734034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:32.734034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:32.734034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:32.734034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:32.734034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:32.734034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:32.734034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:32.734034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:32.734034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:32.734034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:32.734034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:32.734034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:45:32.734034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:45:32.734034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:45:32.734034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:45:32.735296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:32.735296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:32.735296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:32.735296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:32.735296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:32.735296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:32.735296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:32.735296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:32.735296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:32.735296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:32.735296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:32.735296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:32.735296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:32.735296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:32.735296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:32.735296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:32.735296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:45:32.736545 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:45:32.736545 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:45:32.736545 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:32.739013 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 421/822] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:45:32.741452 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aacc6a8-c787-40cc-8982-824abe2ec3ac None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:32.766395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9df407-0321-4e3d-a295-1b47953d0921 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:32.766607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9df407-0321-4e3d-a295-1b47953d0921 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:32.766925 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9df407-0321-4e3d-a295-1b47953d0921 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:32.768523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 422/823] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:45:32.773572 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062f87ae-444c-41f9-9e7c-d4699216cc01 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:32.773858 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062f87ae-444c-41f9-9e7c-d4699216cc01 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:32.774137 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062f87ae-444c-41f9-9e7c-d4699216cc01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:32.792308 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-062f87ae-444c-41f9-9e7c-d4699216cc01 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:32.793528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-062f87ae-444c-41f9-9e7c-d4699216cc01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:32.844867 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aacc6a8-c787-40cc-8982-824abe2ec3ac None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:32.862313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 423/824] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:45:32.880320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1e7b8a-c923-4280-a6eb-d4ea53adc9e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:32.880998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1e7b8a-c923-4280-a6eb-d4ea53adc9e9 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:32.881280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1e7b8a-c923-4280-a6eb-d4ea53adc9e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:32.881553 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1e7b8a-c923-4280-a6eb-d4ea53adc9e9 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:32.882195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df1e7b8a-c923-4280-a6eb-d4ea53adc9e9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:32.882497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df1e7b8a-c923-4280-a6eb-d4ea53adc9e9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:32.883920 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df1e7b8a-c923-4280-a6eb-d4ea53adc9e9 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:32.896930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 402/825] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:45:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:32.902471 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062f87ae-444c-41f9-9e7c-d4699216cc01 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:32.917797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:33.004052 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062f87ae-444c-41f9-9e7c-d4699216cc01 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:33.039727 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6d98e3-e2df-4d1f-9667-a364366752b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:33.040308 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6d98e3-e2df-4d1f-9667-a364366752b6 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:33.040576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6d98e3-e2df-4d1f-9667-a364366752b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:33.040849 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6d98e3-e2df-4d1f-9667-a364366752b6 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:33.048307 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6d98e3-e2df-4d1f-9667-a364366752b6 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:33.049780 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6d98e3-e2df-4d1f-9667-a364366752b6 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:33.062303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-3e6d98e3-e2df-4d1f-9667-a364366752b6 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 16 00:45:33.062303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:33.062303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:33.062303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:33.062303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:33.062303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:33.062303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:33.062303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:33.062303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:33.062303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:33.062303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:33.062303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:33.062303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:45:33.062303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:45:33.062303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:45:33.062303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:33.063472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:33.064917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:33.064917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:33.064917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:45:33.064917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:45:33.064917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:45:33.064917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:45:33.064917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:45:33.064917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:45:33.064917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 16 00:45:33.064917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:33.064917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 424/826] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Oct 16 00:45:32 2022] GET /identity/v3/users/nova => generated 82 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:33.069427 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1e7b8a-c923-4280-a6eb-d4ea53adc9e9 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:33.130699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:33.134267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6d98e3-e2df-4d1f-9667-a364366752b6 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:33.174262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1e7b8a-c923-4280-a6eb-d4ea53adc9e9 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:33.208805 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed71146-d4f6-4f15-a9bf-2cdf3effd477 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:33.210088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed71146-d4f6-4f15-a9bf-2cdf3effd477 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:33.210366 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed71146-d4f6-4f15-a9bf-2cdf3effd477 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:33.210637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed71146-d4f6-4f15-a9bf-2cdf3effd477 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:33.211405 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed71146-d4f6-4f15-a9bf-2cdf3effd477 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:33.211709 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed71146-d4f6-4f15-a9bf-2cdf3effd477 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:33.212625 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:33.213187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed71146-d4f6-4f15-a9bf-2cdf3effd477 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:33.234489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 403/827] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Oct 16 00:45:33 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:33.240484 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed71146-d4f6-4f15-a9bf-2cdf3effd477 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:33.248477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6d98e3-e2df-4d1f-9667-a364366752b6 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:33.284935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398d6eb5-51a7-400d-b6a9-0c58e208a550 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:33.285600 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398d6eb5-51a7-400d-b6a9-0c58e208a550 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:33.285882 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398d6eb5-51a7-400d-b6a9-0c58e208a550 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:33.286158 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398d6eb5-51a7-400d-b6a9-0c58e208a550 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:33.298777 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398d6eb5-51a7-400d-b6a9-0c58e208a550 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:45:33.304511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-398d6eb5-51a7-400d-b6a9-0c58e208a550 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:45:33.304511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:33.304511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:33.304511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:33.304511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:33.304511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:33.304511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:33.304511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:33.304511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:33.304511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:33.304511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:33.304511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:33.304511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:45:33.304511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:45:33.304511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:45:33.304511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:45:33.305767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:45:33.305767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:45:33.305767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:45:33.305767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:45:33.305767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:33.305767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:33.305767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:33.305767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:33.305767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:33.305767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:33.305767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:33.305767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:33.305767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:33.305767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:33.305767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:33.305767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:33.306830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:33.306830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:33.306830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:33.306830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:33.306830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:45:33.306830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:45:33.306830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:45:33.306830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:45:33.306830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:45:33.306830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:45:33.306830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:45:33.306830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:33.306830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 425/828] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:45:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:45:33.312057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398d6eb5-51a7-400d-b6a9-0c58e208a550 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:33.350145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed71146-d4f6-4f15-a9bf-2cdf3effd477 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:33.367287 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:33.389309 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90a344b-f826-4c25-bcd1-540c25820cf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:33.389945 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90a344b-f826-4c25-bcd1-540c25820cf2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:33.390226 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90a344b-f826-4c25-bcd1-540c25820cf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:33.390498 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90a344b-f826-4c25-bcd1-540c25820cf2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:33.421775 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398d6eb5-51a7-400d-b6a9-0c58e208a550 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:33.456619 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:33.463373 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c11bfb8-41de-4f31-a7a8-dc70d29494be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:33.464013 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c11bfb8-41de-4f31-a7a8-dc70d29494be None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:33.464523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c11bfb8-41de-4f31-a7a8-dc70d29494be None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:33.464991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c11bfb8-41de-4f31-a7a8-dc70d29494be None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:33.465677 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c11bfb8-41de-4f31-a7a8-dc70d29494be None admin] RBAC: Adding query filter params (name=admin) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:33.465989 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c11bfb8-41de-4f31-a7a8-dc70d29494be None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:33.466586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90a344b-f826-4c25-bcd1-540c25820cf2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:33.467435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c11bfb8-41de-4f31-a7a8-dc70d29494be None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:33.468410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90a344b-f826-4c25-bcd1-540c25820cf2 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:33.477680 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 426/829] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:45:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:33.484501 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c11bfb8-41de-4f31-a7a8-dc70d29494be None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:33.577036 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 404/830] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:45:33 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 16 00:45:33.583065 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90a344b-f826-4c25-bcd1-540c25820cf2 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:33.600548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:33.618461 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c11bfb8-41de-4f31-a7a8-dc70d29494be None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:33.660047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f556c040-65c5-497e-b687-85fc3a48ec1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:33.660692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f556c040-65c5-497e-b687-85fc3a48ec1f None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:33.660978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f556c040-65c5-497e-b687-85fc3a48ec1f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:33.661253 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f556c040-65c5-497e-b687-85fc3a48ec1f None admin] PATH_INFO: `/v3/users/c5996cb0dc9843c6aa98fb07e2cd740c` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:33.662923 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f556c040-65c5-497e-b687-85fc3a48ec1f None admin] RBAC: Authorizing `identity:get_user(user_id=c5996cb0dc9843c6aa98fb07e2cd740c)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:33.664576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f556c040-65c5-497e-b687-85fc3a48ec1f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:33.666174 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 427/831] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:45:33 2022] GET /identity/v3/users/c5996cb0dc9843c6aa98fb07e2cd740c => generated 285 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:33.672156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f556c040-65c5-497e-b687-85fc3a48ec1f None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:33.686597 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90a344b-f826-4c25-bcd1-540c25820cf2 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:33.721901 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e6e8aa-df4c-45ae-8ddf-a0c853823a0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:33.722537 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e6e8aa-df4c-45ae-8ddf-a0c853823a0d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:33.722816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e6e8aa-df4c-45ae-8ddf-a0c853823a0d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:33.723116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e6e8aa-df4c-45ae-8ddf-a0c853823a0d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:33.733750 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e6e8aa-df4c-45ae-8ddf-a0c853823a0d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:33.734597 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:33.735239 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e6e8aa-df4c-45ae-8ddf-a0c853823a0d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:33.750521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-84e6e8aa-df4c-45ae-8ddf-a0c853823a0d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:45:33.750521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:33.750521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:33.750521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:33.750521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:33.750521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:33.750521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:33.750521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:33.750521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:33.750521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:33.750521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:33.750521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:33.750521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:45:33.750521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:45:33.750521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:45:33.750521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:45:33.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:33.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:33.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:33.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:33.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:33.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:33.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:33.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:33.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:33.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:33.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:33.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:33.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:33.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:33.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:33.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:33.751572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:45:33.753025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:45:33.753025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:45:33.753025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:33.753025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 405/832] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:45:33.757441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e6e8aa-df4c-45ae-8ddf-a0c853823a0d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:33.775565 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f556c040-65c5-497e-b687-85fc3a48ec1f None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:33.818127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f671e998-5ad2-4577-b9c3-a3377cfaf500 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:33.819743 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f671e998-5ad2-4577-b9c3-a3377cfaf500 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:33.820246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f671e998-5ad2-4577-b9c3-a3377cfaf500 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:33.820710 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f671e998-5ad2-4577-b9c3-a3377cfaf500 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:33.824831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:33.863779 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e6e8aa-df4c-45ae-8ddf-a0c853823a0d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:33.895655 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f671e998-5ad2-4577-b9c3-a3377cfaf500 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:33.898024 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f671e998-5ad2-4577-b9c3-a3377cfaf500 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:33.900555 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040252bb-3b3f-475f-a13d-f818e29155a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:33.901276 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040252bb-3b3f-475f-a13d-f818e29155a5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:33.901430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040252bb-3b3f-475f-a13d-f818e29155a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:33.901709 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040252bb-3b3f-475f-a13d-f818e29155a5 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:33.902353 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040252bb-3b3f-475f-a13d-f818e29155a5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:33.902658 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040252bb-3b3f-475f-a13d-f818e29155a5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:33.904180 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040252bb-3b3f-475f-a13d-f818e29155a5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:33.917770 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 406/833] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:45:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:33.923958 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040252bb-3b3f-475f-a13d-f818e29155a5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:33.965055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:34.009559 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 428/834] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:45:33 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 16 00:45:34.015785 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f671e998-5ad2-4577-b9c3-a3377cfaf500 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:34.041876 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040252bb-3b3f-475f-a13d-f818e29155a5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:34.076465 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:34.088880 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67779085-7837-4226-b4ff-e796bb2c53e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:34.089697 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67779085-7837-4226-b4ff-e796bb2c53e5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:34.089982 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67779085-7837-4226-b4ff-e796bb2c53e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:34.090260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67779085-7837-4226-b4ff-e796bb2c53e5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:34.097648 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67779085-7837-4226-b4ff-e796bb2c53e5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:34.099149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67779085-7837-4226-b4ff-e796bb2c53e5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:34.111266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-67779085-7837-4226-b4ff-e796bb2c53e5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:45:34.111266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:34.111266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:34.111266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:34.111266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:34.111266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:34.111266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:34.111266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:34.111266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:34.111266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:34.111266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:34.111266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:34.111266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:45:34.111266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:45:34.111266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:45:34.112579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:45:34.112579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:34.112579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:34.112579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:34.112579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:34.112579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:34.112579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:34.112579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:34.112579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:34.112579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:34.112579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:34.112579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:34.112579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:34.112579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:34.112579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:34.112579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:34.112579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:34.113897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:45:34.113897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:45:34.113897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:45:34.113897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:45:34.113897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:45:34.113897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:45:34.113897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:45:34.113897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:34.113897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 407/835] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:45:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:45:34.119472 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67779085-7837-4226-b4ff-e796bb2c53e5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:34.135294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f671e998-5ad2-4577-b9c3-a3377cfaf500 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:34.172432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4169fe-c228-4170-bd04-10ffba6ff86d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:34.173087 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4169fe-c228-4170-bd04-10ffba6ff86d None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:34.173371 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4169fe-c228-4170-bd04-10ffba6ff86d None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:34.173650 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4169fe-c228-4170-bd04-10ffba6ff86d None admin] PATH_INFO: `/v3/projects/d6bc095e9a044f6c8aa08c2ba3d79489` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:34.174804 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4169fe-c228-4170-bd04-10ffba6ff86d None admin] RBAC: Authorizing `identity:get_project(project_id=d6bc095e9a044f6c8aa08c2ba3d79489)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:34.176281 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4169fe-c228-4170-bd04-10ffba6ff86d None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:34.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 429/836] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:45:34 2022] GET /identity/v3/projects/d6bc095e9a044f6c8aa08c2ba3d79489 => generated 311 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:34.185327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4169fe-c228-4170-bd04-10ffba6ff86d None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:34.219702 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:34.225319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67779085-7837-4226-b4ff-e796bb2c53e5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:34.263539 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8d8ea1-87cb-4a4b-934e-7a85230cc9d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:34.264152 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8d8ea1-87cb-4a4b-934e-7a85230cc9d7 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:34.264428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8d8ea1-87cb-4a4b-934e-7a85230cc9d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:34.264702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8d8ea1-87cb-4a4b-934e-7a85230cc9d7 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:34.265392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8d8ea1-87cb-4a4b-934e-7a85230cc9d7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:34.265693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8d8ea1-87cb-4a4b-934e-7a85230cc9d7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:34.267162 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8d8ea1-87cb-4a4b-934e-7a85230cc9d7 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:34.285720 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 408/837] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:45:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:34.288988 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8d8ea1-87cb-4a4b-934e-7a85230cc9d7 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:34.304882 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4169fe-c228-4170-bd04-10ffba6ff86d None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:34.341292 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:34.347342 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afeb9578-7397-455d-96e9-ef8a0ce250cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:34.348062 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afeb9578-7397-455d-96e9-ef8a0ce250cb None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:34.348417 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afeb9578-7397-455d-96e9-ef8a0ce250cb None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:34.348713 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afeb9578-7397-455d-96e9-ef8a0ce250cb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:34.349533 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afeb9578-7397-455d-96e9-ef8a0ce250cb None admin] RBAC: Adding query filter params (role.id=c5cd777d70a649028f970fd986dc484f, scope.project.id=d6bc095e9a044f6c8aa08c2ba3d79489, user.id=c5996cb0dc9843c6aa98fb07e2cd740c) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:34.349843 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afeb9578-7397-455d-96e9-ef8a0ce250cb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:34.351365 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afeb9578-7397-455d-96e9-ef8a0ce250cb None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:34.359598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 430/838] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:45:34 2022] GET /identity/v3/role_assignments?user.id=c5996cb0dc9843c6aa98fb07e2cd740c&scope.project.id=d6bc095e9a044f6c8aa08c2ba3d79489&role.id=c5cd777d70a649028f970fd986dc484f => generated 608 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:34.408619 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8d8ea1-87cb-4a4b-934e-7a85230cc9d7 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:34.454325 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1b1036-fc25-49a0-a9aa-cb1ea40a64ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:34.455247 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1b1036-fc25-49a0-a9aa-cb1ea40a64ed None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:34.455532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1b1036-fc25-49a0-a9aa-cb1ea40a64ed None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:34.455811 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1b1036-fc25-49a0-a9aa-cb1ea40a64ed None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:34.456571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1b1036-fc25-49a0-a9aa-cb1ea40a64ed None admin] RBAC: Adding query filter params (role.id=7c38bf8daea94adcaaffae01c4253da9, scope.project.id=7deb2c3dae214df7895b40e7339bd6ea, user.id=4eba26813e154595a12a67fe4be5eeff) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:34.456887 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1b1036-fc25-49a0-a9aa-cb1ea40a64ed None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:34.458380 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1b1036-fc25-49a0-a9aa-cb1ea40a64ed None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:34.465439 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 409/839] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:45:34 2022] GET /identity/v3/role_assignments?user.id=4eba26813e154595a12a67fe4be5eeff&scope.project.id=7deb2c3dae214df7895b40e7339bd6ea&role.id=7c38bf8daea94adcaaffae01c4253da9 => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:36.583032 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdfa0c7-2cf9-4289-9e39-ed7e076319b3 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:36.583606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdfa0c7-2cf9-4289-9e39-ed7e076319b3 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:36.583606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdfa0c7-2cf9-4289-9e39-ed7e076319b3 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:36.585845 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 431/840] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:36.591604 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea8120b-015a-409b-a432-ccad6738b53f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:36.592095 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea8120b-015a-409b-a432-ccad6738b53f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:36.592203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea8120b-015a-409b-a432-ccad6738b53f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:36.613064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-dea8120b-015a-409b-a432-ccad6738b53f None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:36.614292 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-dea8120b-015a-409b-a432-ccad6738b53f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:36.688443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 410/841] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:36.960882 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdfa0c7-2cf9-4289-9e39-ed7e076319b3 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:37.058799 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:37.067270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdfa0c7-2cf9-4289-9e39-ed7e076319b3 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:37.109028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f643270b-2405-4ff7-89f4-a6cd42ee899f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:37.109670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f643270b-2405-4ff7-89f4-a6cd42ee899f None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:37.109953 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f643270b-2405-4ff7-89f4-a6cd42ee899f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:37.110228 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f643270b-2405-4ff7-89f4-a6cd42ee899f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:37.111404 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f643270b-2405-4ff7-89f4-a6cd42ee899f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:37.112879 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f643270b-2405-4ff7-89f4-a6cd42ee899f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:37.114489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 432/842] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:45:37.120897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea8120b-015a-409b-a432-ccad6738b53f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:37.164434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:37.220257 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea8120b-015a-409b-a432-ccad6738b53f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:37.369537 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b31d4f9-a59f-481b-a438-efed21ee1fea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:37.370148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b31d4f9-a59f-481b-a438-efed21ee1fea None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:37.370431 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b31d4f9-a59f-481b-a438-efed21ee1fea None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:37.370703 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b31d4f9-a59f-481b-a438-efed21ee1fea None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:37.371388 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b31d4f9-a59f-481b-a438-efed21ee1fea None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:37.372831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b31d4f9-a59f-481b-a438-efed21ee1fea None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:37.394170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 411/843] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:45:37 2022] POST /identity/v3/users => generated 286 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:45:39.592729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49beeb81-34fd-4517-9d38-5c4cb74c3e99 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:39.593253 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49beeb81-34fd-4517-9d38-5c4cb74c3e99 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:39.593370 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49beeb81-34fd-4517-9d38-5c4cb74c3e99 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:39.594620 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 433/844] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:39 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:39.600918 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78674130-46da-4145-bda1-3a16ab52ef9b None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:39.601379 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78674130-46da-4145-bda1-3a16ab52ef9b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:39.601471 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78674130-46da-4145-bda1-3a16ab52ef9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:39.619427 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-78674130-46da-4145-bda1-3a16ab52ef9b None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:39.620644 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-78674130-46da-4145-bda1-3a16ab52ef9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:39.684620 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 412/845] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:39.950582 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49beeb81-34fd-4517-9d38-5c4cb74c3e99 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:40.050300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49beeb81-34fd-4517-9d38-5c4cb74c3e99 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:40.084837 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec56e32-64ea-45d8-94fe-053fadc728ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:40.085472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec56e32-64ea-45d8-94fe-053fadc728ab None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:40.085756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec56e32-64ea-45d8-94fe-053fadc728ab None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:40.086032 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec56e32-64ea-45d8-94fe-053fadc728ab None admin] PATH_INFO: `/v3/roles/d9dcb4cfc0a245a98a7d96c100a497a1` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:40.087469 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec56e32-64ea-45d8-94fe-053fadc728ab None admin] RBAC: Authorizing `identity:get_role(role_id=d9dcb4cfc0a245a98a7d96c100a497a1)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:40.088858 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec56e32-64ea-45d8-94fe-053fadc728ab None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:40.090203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 434/846] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:45:39 2022] GET /identity/v3/roles/d9dcb4cfc0a245a98a7d96c100a497a1 => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:40.096044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78674130-46da-4145-bda1-3a16ab52ef9b None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:40.123274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:40.193969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78674130-46da-4145-bda1-3a16ab52ef9b None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:40.231841 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5910253c-3c1e-4fea-ac95-41cee2ef74fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:40.232390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5910253c-3c1e-4fea-ac95-41cee2ef74fe None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:40.232669 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5910253c-3c1e-4fea-ac95-41cee2ef74fe None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:40.232942 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5910253c-3c1e-4fea-ac95-41cee2ef74fe None admin] PATH_INFO: `/v3/users/4ed9e8e606b84cfeb6287062cc00d7ad` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:40.246045 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5910253c-3c1e-4fea-ac95-41cee2ef74fe None admin] RBAC: Authorizing `identity:get_user(user_id=4ed9e8e606b84cfeb6287062cc00d7ad)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:40.247534 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5910253c-3c1e-4fea-ac95-41cee2ef74fe None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:40.249135 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 413/847] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:45:40 2022] GET /identity/v3/users/4ed9e8e606b84cfeb6287062cc00d7ad => generated 286 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:40.255014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec56e32-64ea-45d8-94fe-053fadc728ab None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:40.285581 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:40.359483 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec56e32-64ea-45d8-94fe-053fadc728ab None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:40.514829 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c769231-411c-46ba-844a-8e3ac8f324f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:40.515413 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c769231-411c-46ba-844a-8e3ac8f324f1 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:40.515707 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c769231-411c-46ba-844a-8e3ac8f324f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:40.515962 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c769231-411c-46ba-844a-8e3ac8f324f1 None admin] PATH_INFO: `/v3/projects/d6bc095e9a044f6c8aa08c2ba3d79489` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:40.517173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c769231-411c-46ba-844a-8e3ac8f324f1 None admin] RBAC: Authorizing `identity:get_project(project_id=d6bc095e9a044f6c8aa08c2ba3d79489)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:40.518598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c769231-411c-46ba-844a-8e3ac8f324f1 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:40.520247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 435/848] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:45:40 2022] GET /identity/v3/projects/d6bc095e9a044f6c8aa08c2ba3d79489 => generated 311 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:40.527309 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5910253c-3c1e-4fea-ac95-41cee2ef74fe None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:40.613376 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:40.630036 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5910253c-3c1e-4fea-ac95-41cee2ef74fe None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:40.663394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdd27b9-e97f-46d8-b538-1abfbd1f5301 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:40.664049 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdd27b9-e97f-46d8-b538-1abfbd1f5301 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:40.664333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdd27b9-e97f-46d8-b538-1abfbd1f5301 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:40.664609 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdd27b9-e97f-46d8-b538-1abfbd1f5301 None admin] PATH_INFO: `/v3/projects/d6bc095e9a044f6c8aa08c2ba3d79489/users/4ed9e8e606b84cfeb6287062cc00d7ad/roles/d9dcb4cfc0a245a98a7d96c100a497a1` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:40.666732 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bdd27b9-e97f-46d8-b538-1abfbd1f5301 None admin] RBAC: Authorizing `identity:create_grant(project_id=d6bc095e9a044f6c8aa08c2ba3d79489, user_id=4ed9e8e606b84cfeb6287062cc00d7ad, role_id=d9dcb4cfc0a245a98a7d96c100a497a1)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:40.668200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bdd27b9-e97f-46d8-b538-1abfbd1f5301 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:40.681613 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 414/849] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:45:40 2022] PUT /identity/v3/projects/d6bc095e9a044f6c8aa08c2ba3d79489/users/4ed9e8e606b84cfeb6287062cc00d7ad/roles/d9dcb4cfc0a245a98a7d96c100a497a1 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:45:42.852662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11090a2a-1edc-46fc-9563-5365d22f55f4 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:42.853197 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11090a2a-1edc-46fc-9563-5365d22f55f4 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:42.853197 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11090a2a-1edc-46fc-9563-5365d22f55f4 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:42.854566 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 436/850] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:42.859797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf31bbe-f1ea-42ab-85af-8678608f9916 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:42.860228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf31bbe-f1ea-42ab-85af-8678608f9916 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:42.860329 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf31bbe-f1ea-42ab-85af-8678608f9916 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:42.878895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-acf31bbe-f1ea-42ab-85af-8678608f9916 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:42.880095 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-acf31bbe-f1ea-42ab-85af-8678608f9916 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:42.946425 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 415/851] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:43.222800 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11090a2a-1edc-46fc-9563-5365d22f55f4 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:43.318376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:43.326251 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11090a2a-1edc-46fc-9563-5365d22f55f4 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:43.383784 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cc304a-1f17-462b-942d-7ea662162c2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:43.384852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cc304a-1f17-462b-942d-7ea662162c2d None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:43.385360 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cc304a-1f17-462b-942d-7ea662162c2d None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:43.385866 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cc304a-1f17-462b-942d-7ea662162c2d None admin] PATH_INFO: `/v3/roles/d9dcb4cfc0a245a98a7d96c100a497a1` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:43.388227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cc304a-1f17-462b-942d-7ea662162c2d None admin] RBAC: Authorizing `identity:get_role(role_id=d9dcb4cfc0a245a98a7d96c100a497a1)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:43.390879 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cc304a-1f17-462b-942d-7ea662162c2d None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:43.393295 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 437/852] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:45:43 2022] GET /identity/v3/roles/d9dcb4cfc0a245a98a7d96c100a497a1 => generated 223 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:43.399709 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf31bbe-f1ea-42ab-85af-8678608f9916 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:43.479936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:43.502971 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf31bbe-f1ea-42ab-85af-8678608f9916 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:43.537336 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c66c6b-9796-4708-b86f-81ed921210a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:43.537959 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c66c6b-9796-4708-b86f-81ed921210a6 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:43.538233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c66c6b-9796-4708-b86f-81ed921210a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:43.538502 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c66c6b-9796-4708-b86f-81ed921210a6 None admin] PATH_INFO: `/v3/users/4ed9e8e606b84cfeb6287062cc00d7ad` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:43.539626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c66c6b-9796-4708-b86f-81ed921210a6 None admin] RBAC: Authorizing `identity:get_user(user_id=4ed9e8e606b84cfeb6287062cc00d7ad)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:43.541050 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c66c6b-9796-4708-b86f-81ed921210a6 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:43.542593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 416/853] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:45:43 2022] GET /identity/v3/users/4ed9e8e606b84cfeb6287062cc00d7ad => generated 286 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:43.548479 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cc304a-1f17-462b-942d-7ea662162c2d None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:43.647658 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cc304a-1f17-462b-942d-7ea662162c2d None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:43.680045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326333a4-42a1-4044-bbdf-b96ee171e83c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:43.680641 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326333a4-42a1-4044-bbdf-b96ee171e83c None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:43.680917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326333a4-42a1-4044-bbdf-b96ee171e83c None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:43.681183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326333a4-42a1-4044-bbdf-b96ee171e83c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:43.713500 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:43.752755 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326333a4-42a1-4044-bbdf-b96ee171e83c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:43.754143 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326333a4-42a1-4044-bbdf-b96ee171e83c None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:43.851132 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 438/854] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:45:43 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 16 00:45:43.857679 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c66c6b-9796-4708-b86f-81ed921210a6 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:43.956448 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c66c6b-9796-4708-b86f-81ed921210a6 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:43.991351 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b629a36-68d2-4377-a72e-0d6a1cb6f0e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:43.991986 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b629a36-68d2-4377-a72e-0d6a1cb6f0e0 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:43.992267 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b629a36-68d2-4377-a72e-0d6a1cb6f0e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:43.992540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b629a36-68d2-4377-a72e-0d6a1cb6f0e0 None admin] PATH_INFO: `/v3/projects/d6bc095e9a044f6c8aa08c2ba3d79489` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:43.993657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b629a36-68d2-4377-a72e-0d6a1cb6f0e0 None admin] RBAC: Authorizing `identity:get_project(project_id=d6bc095e9a044f6c8aa08c2ba3d79489)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:43.995122 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b629a36-68d2-4377-a72e-0d6a1cb6f0e0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:43.996758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 417/855] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:45:43 2022] GET /identity/v3/projects/d6bc095e9a044f6c8aa08c2ba3d79489 => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:44.003653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326333a4-42a1-4044-bbdf-b96ee171e83c None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:44.070210 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:44.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326333a4-42a1-4044-bbdf-b96ee171e83c None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:44.139196 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eea10e2-b456-4290-ac4b-52b46cccdb8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:44.139801 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eea10e2-b456-4290-ac4b-52b46cccdb8c None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:44.140076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eea10e2-b456-4290-ac4b-52b46cccdb8c None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:44.140345 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eea10e2-b456-4290-ac4b-52b46cccdb8c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:44.141063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eea10e2-b456-4290-ac4b-52b46cccdb8c None admin] RBAC: Adding query filter params (role.id=d9dcb4cfc0a245a98a7d96c100a497a1, scope.project.id=d6bc095e9a044f6c8aa08c2ba3d79489, user.id=4ed9e8e606b84cfeb6287062cc00d7ad) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:44.141364 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eea10e2-b456-4290-ac4b-52b46cccdb8c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:44.142783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eea10e2-b456-4290-ac4b-52b46cccdb8c None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:44.149291 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 439/856] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:45:44 2022] GET /identity/v3/role_assignments?user.id=4ed9e8e606b84cfeb6287062cc00d7ad&scope.project.id=d6bc095e9a044f6c8aa08c2ba3d79489&role.id=d9dcb4cfc0a245a98a7d96c100a497a1 => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:46.332094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602f4985-4193-4fd6-8845-4d3b7447eda9 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:46.332622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602f4985-4193-4fd6-8845-4d3b7447eda9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:46.332622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602f4985-4193-4fd6-8845-4d3b7447eda9 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:46.334769 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 418/857] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:46.340111 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2cc9bd-3c84-4185-b540-5ed4c3926f9b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:46.340510 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2cc9bd-3c84-4185-b540-5ed4c3926f9b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:46.340660 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2cc9bd-3c84-4185-b540-5ed4c3926f9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:46.359707 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-ac2cc9bd-3c84-4185-b540-5ed4c3926f9b None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:46.360927 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-ac2cc9bd-3c84-4185-b540-5ed4c3926f9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:46.431302 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 440/858] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:46.699992 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602f4985-4193-4fd6-8845-4d3b7447eda9 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:46.715093 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:46.801158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602f4985-4193-4fd6-8845-4d3b7447eda9 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:46.834082 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12e4d48-d171-48fe-977d-216a6bcc6423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:46.834703 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12e4d48-d171-48fe-977d-216a6bcc6423 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:46.835010 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12e4d48-d171-48fe-977d-216a6bcc6423 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:46.835278 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12e4d48-d171-48fe-977d-216a6bcc6423 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:46.836365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12e4d48-d171-48fe-977d-216a6bcc6423 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:46.837791 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12e4d48-d171-48fe-977d-216a6bcc6423 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:46.839373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 419/859] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:46.845194 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2cc9bd-3c84-4185-b540-5ed4c3926f9b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:46.903592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:46.944610 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2cc9bd-3c84-4185-b540-5ed4c3926f9b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:46.979287 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1302de72-55de-4dcc-a711-1f5826a28730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:46.979821 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1302de72-55de-4dcc-a711-1f5826a28730 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:46.980103 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1302de72-55de-4dcc-a711-1f5826a28730 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:46.980373 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1302de72-55de-4dcc-a711-1f5826a28730 None admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:46.981033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1302de72-55de-4dcc-a711-1f5826a28730 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:46.982462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1302de72-55de-4dcc-a711-1f5826a28730 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:47.008960 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 441/860] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 16 00:45:46 2022] POST /identity/v3/projects => generated 311 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:45:49.191574 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de4fa4f-d27c-47a3-8775-9b5b1e199d43 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:49.192115 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de4fa4f-d27c-47a3-8775-9b5b1e199d43 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:49.192115 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de4fa4f-d27c-47a3-8775-9b5b1e199d43 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:49.193490 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 420/861] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:49.198799 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea5080e-825f-4813-be9c-e9dd0ca1924e None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:49.199211 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea5080e-825f-4813-be9c-e9dd0ca1924e None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:49.199384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea5080e-825f-4813-be9c-e9dd0ca1924e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:49.218505 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-6ea5080e-825f-4813-be9c-e9dd0ca1924e None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:49.219866 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6ea5080e-825f-4813-be9c-e9dd0ca1924e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:49.285684 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 442/862] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:49.554101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de4fa4f-d27c-47a3-8775-9b5b1e199d43 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:49.636390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:49.657174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de4fa4f-d27c-47a3-8775-9b5b1e199d43 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:49.690106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8031d9f7-fb28-4832-aefd-9b25915f6080 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:49.690734 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8031d9f7-fb28-4832-aefd-9b25915f6080 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:49.691043 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8031d9f7-fb28-4832-aefd-9b25915f6080 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:49.691321 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8031d9f7-fb28-4832-aefd-9b25915f6080 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:49.692425 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8031d9f7-fb28-4832-aefd-9b25915f6080 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:49.694359 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8031d9f7-fb28-4832-aefd-9b25915f6080 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:49.695945 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 421/863] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:45:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:49.701760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea5080e-825f-4813-be9c-e9dd0ca1924e None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:49.744797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:49.805008 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea5080e-825f-4813-be9c-e9dd0ca1924e None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:49.841725 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe8708e-85ad-49bf-a61d-9e24a4e80eca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:49.842332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe8708e-85ad-49bf-a61d-9e24a4e80eca None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:49.842613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe8708e-85ad-49bf-a61d-9e24a4e80eca None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:49.842908 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe8708e-85ad-49bf-a61d-9e24a4e80eca None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:49.843559 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe8708e-85ad-49bf-a61d-9e24a4e80eca None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:49.844994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe8708e-85ad-49bf-a61d-9e24a4e80eca None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:49.867032 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 443/864] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:45:49 2022] POST /identity/v3/users => generated 286 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:45:52.054800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a382eeb5-2abc-4e55-b314-10dcf4ab0463 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:52.055392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a382eeb5-2abc-4e55-b314-10dcf4ab0463 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:52.055392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a382eeb5-2abc-4e55-b314-10dcf4ab0463 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:52.056713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 422/865] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:52.061994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1fbec6-cc63-441e-b82b-d181f50935ca None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:52.062423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1fbec6-cc63-441e-b82b-d181f50935ca None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:52.062552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1fbec6-cc63-441e-b82b-d181f50935ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:52.080729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-4d1fbec6-cc63-441e-b82b-d181f50935ca None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:52.081923 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4d1fbec6-cc63-441e-b82b-d181f50935ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:52.147846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 444/866] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:52.418074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a382eeb5-2abc-4e55-b314-10dcf4ab0463 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:52.521333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a382eeb5-2abc-4e55-b314-10dcf4ab0463 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:52.555585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781a0ba3-6b2e-4bf8-ac00-b19be11585e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:52.556216 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781a0ba3-6b2e-4bf8-ac00-b19be11585e8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:52.556500 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781a0ba3-6b2e-4bf8-ac00-b19be11585e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:52.556774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781a0ba3-6b2e-4bf8-ac00-b19be11585e8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:52.566566 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781a0ba3-6b2e-4bf8-ac00-b19be11585e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:45:52.571646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-781a0ba3-6b2e-4bf8-ac00-b19be11585e8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:45:52.571646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:52.571646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:52.571646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:52.571646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:52.571646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:52.571646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:52.571646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:52.571646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:52.571646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:52.571646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:52.571646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:52.571646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:45:52.571646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:45:52.571646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:45:52.571646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:45:52.572736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:45:52.572736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:45:52.572736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:45:52.572736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:45:52.572736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:52.572736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:52.572736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:52.572736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:52.572736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:52.572736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:52.572736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:52.572736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:52.572736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:52.572736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:52.572736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:52.572736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:52.573601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:52.573601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:52.573601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:52.573601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:52.573601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:45:52.573601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:45:52.573601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:45:52.573601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:45:52.573601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:45:52.573601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:45:52.573601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:45:52.573601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:45:52.573601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 423/867] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:45:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:52.578845 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1fbec6-cc63-441e-b82b-d181f50935ca None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:52.606467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:52.679175 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1fbec6-cc63-441e-b82b-d181f50935ca None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:52.714829 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a92da0-abcc-437b-8735-1946f39c104a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:52.715477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a92da0-abcc-437b-8735-1946f39c104a None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:52.715759 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a92da0-abcc-437b-8735-1946f39c104a None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:52.716035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a92da0-abcc-437b-8735-1946f39c104a None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:52.716700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a92da0-abcc-437b-8735-1946f39c104a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:52.717009 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a92da0-abcc-437b-8735-1946f39c104a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:52.718377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a92da0-abcc-437b-8735-1946f39c104a None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:52.727021 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 445/868] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:45:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:52.733629 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781a0ba3-6b2e-4bf8-ac00-b19be11585e8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:52.745055 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:52.835362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781a0ba3-6b2e-4bf8-ac00-b19be11585e8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:52.869750 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7961b025-7fcc-4ac4-be69-880010b1103d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:52.870385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7961b025-7fcc-4ac4-be69-880010b1103d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:52.870661 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7961b025-7fcc-4ac4-be69-880010b1103d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:52.870962 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7961b025-7fcc-4ac4-be69-880010b1103d None admin] PATH_INFO: `/v3/users/668a3626b4bf4da4933c4765f4080b6e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:52.884408 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7961b025-7fcc-4ac4-be69-880010b1103d None admin] RBAC: Authorizing `identity:get_user(user_id=668a3626b4bf4da4933c4765f4080b6e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:52.885871 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7961b025-7fcc-4ac4-be69-880010b1103d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:52.887486 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 424/869] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:45:52 2022] GET /identity/v3/users/668a3626b4bf4da4933c4765f4080b6e => generated 286 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:52.893326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a92da0-abcc-437b-8735-1946f39c104a None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:52.982333 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:52.992213 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a92da0-abcc-437b-8735-1946f39c104a None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:53.025072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa4a710-e717-4733-9cab-5528e1023e88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:53.025701 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa4a710-e717-4733-9cab-5528e1023e88 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:53.025983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa4a710-e717-4733-9cab-5528e1023e88 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:53.026254 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa4a710-e717-4733-9cab-5528e1023e88 None admin] PATH_INFO: `/v3/projects/1be1b0ec642447ea810bb8d1be85a8fb` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:53.027391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa4a710-e717-4733-9cab-5528e1023e88 None admin] RBAC: Authorizing `identity:get_project(project_id=1be1b0ec642447ea810bb8d1be85a8fb)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:53.028809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa4a710-e717-4733-9cab-5528e1023e88 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:53.030416 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 446/870] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:45:52 2022] GET /identity/v3/projects/1be1b0ec642447ea810bb8d1be85a8fb => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:53.037263 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7961b025-7fcc-4ac4-be69-880010b1103d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:53.102197 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:53.136254 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7961b025-7fcc-4ac4-be69-880010b1103d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:53.170831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93de56e6-55d5-4740-bb15-bab64128ad60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:53.171514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93de56e6-55d5-4740-bb15-bab64128ad60 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:53.171793 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93de56e6-55d5-4740-bb15-bab64128ad60 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:53.172068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93de56e6-55d5-4740-bb15-bab64128ad60 None admin] PATH_INFO: `/v3/projects/1be1b0ec642447ea810bb8d1be85a8fb/users/668a3626b4bf4da4933c4765f4080b6e/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:53.173948 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93de56e6-55d5-4740-bb15-bab64128ad60 None admin] RBAC: Authorizing `identity:create_grant(project_id=1be1b0ec642447ea810bb8d1be85a8fb, user_id=668a3626b4bf4da4933c4765f4080b6e, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:53.175422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93de56e6-55d5-4740-bb15-bab64128ad60 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:53.189261 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 425/871] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:45:53 2022] PUT /identity/v3/projects/1be1b0ec642447ea810bb8d1be85a8fb/users/668a3626b4bf4da4933c4765f4080b6e/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:45:55.340280 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5656ab15-22ae-42ac-b5d2-8d5cf6db02f7 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:55.340813 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5656ab15-22ae-42ac-b5d2-8d5cf6db02f7 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:55.340813 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5656ab15-22ae-42ac-b5d2-8d5cf6db02f7 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:55.342136 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 447/872] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:55.347319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c620b9-a856-4501-a8f0-206668f7f0c1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:55.347728 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c620b9-a856-4501-a8f0-206668f7f0c1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:55.347891 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c620b9-a856-4501-a8f0-206668f7f0c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:55.365969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a4c620b9-a856-4501-a8f0-206668f7f0c1 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:55.367168 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a4c620b9-a856-4501-a8f0-206668f7f0c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:55.432711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 426/873] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:55.700552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5656ab15-22ae-42ac-b5d2-8d5cf6db02f7 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:55.802109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5656ab15-22ae-42ac-b5d2-8d5cf6db02f7 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:55.835940 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6cf623-6c10-455c-be28-cb19607118c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:55.836579 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6cf623-6c10-455c-be28-cb19607118c8 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:55.836860 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6cf623-6c10-455c-be28-cb19607118c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:55.837135 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6cf623-6c10-455c-be28-cb19607118c8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:55.847486 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6cf623-6c10-455c-be28-cb19607118c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:45:55.852473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-5c6cf623-6c10-455c-be28-cb19607118c8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:45:55.852473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:45:55.852473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:45:55.852473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:45:55.852473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:45:55.852473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:45:55.852473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:45:55.852473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:45:55.852473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:45:55.852473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:45:55.852473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:45:55.852473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:45:55.852473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:45:55.852473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:45:55.852473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:45:55.852473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:45:55.853567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:45:55.853567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:45:55.853567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:45:55.853567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:45:55.853567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:45:55.853567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:45:55.853567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:45:55.853567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:45:55.853567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:45:55.853567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:45:55.853567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:45:55.853567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:45:55.853567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:45:55.853567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:45:55.853567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:45:55.853567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:45:55.854451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:45:55.854451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:45:55.854451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:45:55.854451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:45:55.854451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:45:55.854451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:45:55.854451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:45:55.854451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:45:55.854451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:45:55.854451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:45:55.854451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:45:55.854451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:45:55.854451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 448/874] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:45:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:45:55.859807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c620b9-a856-4501-a8f0-206668f7f0c1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:55.913441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:55.957589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c620b9-a856-4501-a8f0-206668f7f0c1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:55.990161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f442ef9a-88c8-464b-ad80-8a0d89f21101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:55.990752 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f442ef9a-88c8-464b-ad80-8a0d89f21101 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:55.991060 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f442ef9a-88c8-464b-ad80-8a0d89f21101 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:55.991329 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f442ef9a-88c8-464b-ad80-8a0d89f21101 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:55.991979 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f442ef9a-88c8-464b-ad80-8a0d89f21101 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:55.992279 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f442ef9a-88c8-464b-ad80-8a0d89f21101 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:55.993636 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f442ef9a-88c8-464b-ad80-8a0d89f21101 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:56.001379 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 427/875] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:45:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:56.008315 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6cf623-6c10-455c-be28-cb19607118c8 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:56.027560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:56.110382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6cf623-6c10-455c-be28-cb19607118c8 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:56.145562 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50bfe93-13f0-467c-a760-d1249bf64aa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:56.146199 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50bfe93-13f0-467c-a760-d1249bf64aa3 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:56.146484 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50bfe93-13f0-467c-a760-d1249bf64aa3 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:56.146765 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50bfe93-13f0-467c-a760-d1249bf64aa3 None admin] PATH_INFO: `/v3/users/668a3626b4bf4da4933c4765f4080b6e` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:56.147932 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50bfe93-13f0-467c-a760-d1249bf64aa3 None admin] RBAC: Authorizing `identity:get_user(user_id=668a3626b4bf4da4933c4765f4080b6e)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:56.149378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50bfe93-13f0-467c-a760-d1249bf64aa3 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:56.150991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 449/876] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:45:56 2022] GET /identity/v3/users/668a3626b4bf4da4933c4765f4080b6e => generated 286 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:56.156842 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f442ef9a-88c8-464b-ad80-8a0d89f21101 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:56.259012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f442ef9a-88c8-464b-ad80-8a0d89f21101 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:56.275615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:56.293216 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7e9978-34b5-427c-b4c6-b5ce026ee85d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:56.293828 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7e9978-34b5-427c-b4c6-b5ce026ee85d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:56.294110 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7e9978-34b5-427c-b4c6-b5ce026ee85d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:56.294384 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7e9978-34b5-427c-b4c6-b5ce026ee85d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:56.361726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7e9978-34b5-427c-b4c6-b5ce026ee85d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:56.363224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7e9978-34b5-427c-b4c6-b5ce026ee85d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:56.463619 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 428/877] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:45:56 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 16 00:45:56.469572 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50bfe93-13f0-467c-a760-d1249bf64aa3 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:56.558655 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:56.573899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50bfe93-13f0-467c-a760-d1249bf64aa3 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:56.609478 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69cee80-dc8d-427f-846c-b3f02ecc1b4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:56.610108 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69cee80-dc8d-427f-846c-b3f02ecc1b4e None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:56.610390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69cee80-dc8d-427f-846c-b3f02ecc1b4e None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:56.610659 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69cee80-dc8d-427f-846c-b3f02ecc1b4e None admin] PATH_INFO: `/v3/projects/1be1b0ec642447ea810bb8d1be85a8fb` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:56.611798 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69cee80-dc8d-427f-846c-b3f02ecc1b4e None admin] RBAC: Authorizing `identity:get_project(project_id=1be1b0ec642447ea810bb8d1be85a8fb)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:56.613226 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69cee80-dc8d-427f-846c-b3f02ecc1b4e None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:56.615176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 450/878] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:45:56 2022] GET /identity/v3/projects/1be1b0ec642447ea810bb8d1be85a8fb => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:56.621398 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7e9978-34b5-427c-b4c6-b5ce026ee85d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:56.639946 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:45:56.724184 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7e9978-34b5-427c-b4c6-b5ce026ee85d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:56.759707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891903cb-0a84-4f82-918c-b6c7c1b9523b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:56.760307 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891903cb-0a84-4f82-918c-b6c7c1b9523b None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:56.760584 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891903cb-0a84-4f82-918c-b6c7c1b9523b None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:56.760851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891903cb-0a84-4f82-918c-b6c7c1b9523b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:56.761562 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891903cb-0a84-4f82-918c-b6c7c1b9523b None admin] RBAC: Adding query filter params (role.id=c5cd777d70a649028f970fd986dc484f, scope.project.id=1be1b0ec642447ea810bb8d1be85a8fb, user.id=668a3626b4bf4da4933c4765f4080b6e) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:45:56.761865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891903cb-0a84-4f82-918c-b6c7c1b9523b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:56.763349 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891903cb-0a84-4f82-918c-b6c7c1b9523b None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:56.769025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 429/879] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:45:56 2022] GET /identity/v3/role_assignments?user.id=668a3626b4bf4da4933c4765f4080b6e&scope.project.id=1be1b0ec642447ea810bb8d1be85a8fb&role.id=c5cd777d70a649028f970fd986dc484f => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:45:58.929352 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df66154-2c70-4a14-8e85-a44d67cd76d8 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:58.929922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df66154-2c70-4a14-8e85-a44d67cd76d8 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:58.929922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df66154-2c70-4a14-8e85-a44d67cd76d8 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:58.931283 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 451/880] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:45:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:45:58.936793 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d6d0b7-fdc5-4651-b50e-bbc2e2817720 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:58.937237 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d6d0b7-fdc5-4651-b50e-bbc2e2817720 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:58.937373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d6d0b7-fdc5-4651-b50e-bbc2e2817720 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:58.957075 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-22d6d0b7-fdc5-4651-b50e-bbc2e2817720 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:45:58.958488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-22d6d0b7-fdc5-4651-b50e-bbc2e2817720 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:45:59.030193 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 430/881] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:45:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:45:59.303596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df66154-2c70-4a14-8e85-a44d67cd76d8 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:45:59.407386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df66154-2c70-4a14-8e85-a44d67cd76d8 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:45:59.443637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f58b75-5d26-49a0-b662-719379c6b356 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:45:59.444264 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f58b75-5d26-49a0-b662-719379c6b356 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:45:59.444547 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f58b75-5d26-49a0-b662-719379c6b356 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:45:59.444820 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f58b75-5d26-49a0-b662-719379c6b356 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:45:59.445499 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f58b75-5d26-49a0-b662-719379c6b356 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:45:59.446897 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f58b75-5d26-49a0-b662-719379c6b356 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:45:59.464632 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 452/882] 10.222.0.25 () {60 vars in 1138 bytes} [Sun Oct 16 00:45:59 2022] POST /identity/v3/domains => generated 267 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:46:01.777350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463d33fb-f9da-478c-a1ef-3453a9e7f5f9 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:01.777973 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463d33fb-f9da-478c-a1ef-3453a9e7f5f9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:01.777973 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463d33fb-f9da-478c-a1ef-3453a9e7f5f9 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:01.779326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 431/883] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:46:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:46:01.784784 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb707389-0f23-4982-8753-3a1b03d97775 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:01.785236 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb707389-0f23-4982-8753-3a1b03d97775 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:01.785350 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb707389-0f23-4982-8753-3a1b03d97775 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:01.804511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-eb707389-0f23-4982-8753-3a1b03d97775 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:46:01.805708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-eb707389-0f23-4982-8753-3a1b03d97775 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:46:01.814810 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:46:01.873423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 453/884] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:46:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:46:02.139880 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463d33fb-f9da-478c-a1ef-3453a9e7f5f9 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:46:02.155048 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:46:02.245634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463d33fb-f9da-478c-a1ef-3453a9e7f5f9 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:46:02.280775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bf9401-7e07-4c66-b3db-91133709c1a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:46:02.281423 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bf9401-7e07-4c66-b3db-91133709c1a2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:02.281728 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bf9401-7e07-4c66-b3db-91133709c1a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:02.282016 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bf9401-7e07-4c66-b3db-91133709c1a2 None admin] PATH_INFO: `/v3/domains/db46d8ec4e864742b62993d351b8a5c4` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:02.293216 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bf9401-7e07-4c66-b3db-91133709c1a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=db46d8ec4e864742b62993d351b8a5c4)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:46:02.294665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bf9401-7e07-4c66-b3db-91133709c1a2 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:46:02.296465 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 432/885] 10.222.0.25 () {56 vars in 1183 bytes} [Sun Oct 16 00:46:02 2022] GET /identity/v3/domains/db46d8ec4e864742b62993d351b8a5c4 => generated 267 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:46:02.303014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb707389-0f23-4982-8753-3a1b03d97775 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:46:02.401718 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb707389-0f23-4982-8753-3a1b03d97775 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:46:02.439569 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7dfe54b-5113-44c4-b828-461e24b6814b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:46:02.440183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dfe54b-5113-44c4-b828-461e24b6814b None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:02.440459 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dfe54b-5113-44c4-b828-461e24b6814b None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:02.440729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7dfe54b-5113-44c4-b828-461e24b6814b None admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:02.441377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7dfe54b-5113-44c4-b828-461e24b6814b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:46:02.442810 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7dfe54b-5113-44c4-b828-461e24b6814b None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:46:02.457997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:46:02.470881 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 454/886] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 16 00:46:02 2022] POST /identity/v3/projects => generated 361 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:46:04.647326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d466ce-4fb8-4d0a-b620-7796e82543be None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:04.647853 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d466ce-4fb8-4d0a-b620-7796e82543be None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:04.647853 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d466ce-4fb8-4d0a-b620-7796e82543be None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:04.649199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 433/887] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:46:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:46:04.654335 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0611cd-d66a-438b-8c49-7e71a1d2521a None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:04.654794 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0611cd-d66a-438b-8c49-7e71a1d2521a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:04.654924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0611cd-d66a-438b-8c49-7e71a1d2521a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:04.674431 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-6a0611cd-d66a-438b-8c49-7e71a1d2521a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:46:04.675757 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6a0611cd-d66a-438b-8c49-7e71a1d2521a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:46:04.744984 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 455/888] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:46:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:46:05.017496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d466ce-4fb8-4d0a-b620-7796e82543be None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:46:05.090202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:46:05.121380 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d466ce-4fb8-4d0a-b620-7796e82543be None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:46:05.154717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587cd83d-57b3-46a4-a20b-4f97d9405617 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:46:05.155377 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587cd83d-57b3-46a4-a20b-4f97d9405617 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:05.155661 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587cd83d-57b3-46a4-a20b-4f97d9405617 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:05.155939 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587cd83d-57b3-46a4-a20b-4f97d9405617 None admin] PATH_INFO: `/v3/domains/db46d8ec4e864742b62993d351b8a5c4` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:05.157054 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587cd83d-57b3-46a4-a20b-4f97d9405617 None admin] RBAC: Authorizing `identity:get_domain(domain_id=db46d8ec4e864742b62993d351b8a5c4)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:46:05.158493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587cd83d-57b3-46a4-a20b-4f97d9405617 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:46:05.160098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 434/889] 10.222.0.25 () {56 vars in 1183 bytes} [Sun Oct 16 00:46:05 2022] GET /identity/v3/domains/db46d8ec4e864742b62993d351b8a5c4 => generated 267 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:46:05.166435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0611cd-d66a-438b-8c49-7e71a1d2521a None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:46:05.270256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0611cd-d66a-438b-8c49-7e71a1d2521a None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:46:05.295538 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:46:05.305202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988725b3-1bf4-4da5-b877-f8feebb7a9ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:46:05.305818 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988725b3-1bf4-4da5-b877-f8feebb7a9ec None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:05.306095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988725b3-1bf4-4da5-b877-f8feebb7a9ec None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:05.306383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988725b3-1bf4-4da5-b877-f8feebb7a9ec None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:05.307059 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988725b3-1bf4-4da5-b877-f8feebb7a9ec None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:46:05.308492 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988725b3-1bf4-4da5-b877-f8feebb7a9ec None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:46:05.330594 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 456/890] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:46:05 2022] POST /identity/v3/users => generated 311 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:46:07.522440 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a99f6c-2f4e-43f3-9c63-16fe93b95541 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:07.522954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a99f6c-2f4e-43f3-9c63-16fe93b95541 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:07.523062 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a99f6c-2f4e-43f3-9c63-16fe93b95541 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:07.524375 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 435/891] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:46:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:46:07.529827 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab58ab5-ddf2-4010-9d39-bebd5131dc41 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:07.530221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab58ab5-ddf2-4010-9d39-bebd5131dc41 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:07.530367 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab58ab5-ddf2-4010-9d39-bebd5131dc41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:07.548590 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-0ab58ab5-ddf2-4010-9d39-bebd5131dc41 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:46:07.549768 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-0ab58ab5-ddf2-4010-9d39-bebd5131dc41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:46:07.621763 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 457/892] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:46:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:46:07.908949 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a99f6c-2f4e-43f3-9c63-16fe93b95541 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:46:08.011142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a99f6c-2f4e-43f3-9c63-16fe93b95541 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:46:08.043934 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499ff748-07a2-4070-8216-e25be83552df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:46:08.044594 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499ff748-07a2-4070-8216-e25be83552df None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:08.044880 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499ff748-07a2-4070-8216-e25be83552df None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:08.045157 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499ff748-07a2-4070-8216-e25be83552df None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:08.054888 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:46:08.059131 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499ff748-07a2-4070-8216-e25be83552df None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:46:08.064280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-499ff748-07a2-4070-8216-e25be83552df None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:46:08.064280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:46:08.064280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:46:08.064280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:46:08.064280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:46:08.064280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:46:08.064280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:46:08.064280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:46:08.064280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:46:08.064280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:46:08.064280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:46:08.064280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:46:08.064280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:46:08.064280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:46:08.064280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:46:08.064280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:46:08.065894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:46:08.065894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:46:08.065894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:46:08.065894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:46:08.065894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:46:08.065894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:46:08.065894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:46:08.065894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:46:08.065894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:46:08.065894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:46:08.065894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:46:08.065894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:46:08.065894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:46:08.065894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:46:08.065894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:46:08.065894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:46:08.067917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:46:08.067917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:46:08.067917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:46:08.067917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:46:08.067917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:46:08.067917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:46:08.067917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:46:08.067917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:46:08.067917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:46:08.067917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:46:08.067917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:46:08.067917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:46:08.067917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 436/893] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:46:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:46:08.072354 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab58ab5-ddf2-4010-9d39-bebd5131dc41 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:46:08.171843 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab58ab5-ddf2-4010-9d39-bebd5131dc41 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:46:08.186522 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:46:08.209914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e79eeb-d7f1-4399-8e02-6b01c1078cfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:46:08.210536 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e79eeb-d7f1-4399-8e02-6b01c1078cfc None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:08.210818 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e79eeb-d7f1-4399-8e02-6b01c1078cfc None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:08.211125 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e79eeb-d7f1-4399-8e02-6b01c1078cfc None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:08.211790 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e79eeb-d7f1-4399-8e02-6b01c1078cfc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:46:08.212095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e79eeb-d7f1-4399-8e02-6b01c1078cfc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:46:08.213497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e79eeb-d7f1-4399-8e02-6b01c1078cfc None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:46:08.221928 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 458/894] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:46:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:46:08.228396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499ff748-07a2-4070-8216-e25be83552df None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:46:08.333075 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499ff748-07a2-4070-8216-e25be83552df None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:46:08.367193 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5f7df0-ced1-4a41-9e95-cb46bbab1f15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:46:08.367869 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5f7df0-ced1-4a41-9e95-cb46bbab1f15 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:08.368163 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5f7df0-ced1-4a41-9e95-cb46bbab1f15 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:08.368441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5f7df0-ced1-4a41-9e95-cb46bbab1f15 None admin] PATH_INFO: `/v3/users/2a2093be1a4f48acbacb6129749c8ed9` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:08.383348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5f7df0-ced1-4a41-9e95-cb46bbab1f15 None admin] RBAC: Authorizing `identity:get_user(user_id=2a2093be1a4f48acbacb6129749c8ed9)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:46:08.384891 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5f7df0-ced1-4a41-9e95-cb46bbab1f15 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:46:08.386538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 437/895] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:46:08 2022] GET /identity/v3/users/2a2093be1a4f48acbacb6129749c8ed9 => generated 311 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:46:08.392407 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e79eeb-d7f1-4399-8e02-6b01c1078cfc None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:46:08.492917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e79eeb-d7f1-4399-8e02-6b01c1078cfc None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:46:08.526408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f24408-a4ce-49af-8563-9e1242a2e07d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:46:08.527069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f24408-a4ce-49af-8563-9e1242a2e07d None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:08.527354 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f24408-a4ce-49af-8563-9e1242a2e07d None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:08.527629 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f24408-a4ce-49af-8563-9e1242a2e07d None admin] PATH_INFO: `/v3/projects/c77c2e61a3db498a9e2d4fa45be9833a` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:08.528739 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f24408-a4ce-49af-8563-9e1242a2e07d None admin] RBAC: Authorizing `identity:get_project(project_id=c77c2e61a3db498a9e2d4fa45be9833a)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:46:08.530176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f24408-a4ce-49af-8563-9e1242a2e07d None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:46:08.531842 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 459/896] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:46:08 2022] GET /identity/v3/projects/c77c2e61a3db498a9e2d4fa45be9833a => generated 361 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:46:08.538208 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5f7df0-ced1-4a41-9e95-cb46bbab1f15 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:46:08.600306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:46:08.643178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5f7df0-ced1-4a41-9e95-cb46bbab1f15 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:46:08.678447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8540024e-8193-4f86-900e-85337206aadf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:46:08.679204 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8540024e-8193-4f86-900e-85337206aadf None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:08.679572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8540024e-8193-4f86-900e-85337206aadf None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:08.679853 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8540024e-8193-4f86-900e-85337206aadf None admin] PATH_INFO: `/v3/projects/c77c2e61a3db498a9e2d4fa45be9833a/users/2a2093be1a4f48acbacb6129749c8ed9/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:08.681895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8540024e-8193-4f86-900e-85337206aadf None admin] RBAC: Authorizing `identity:create_grant(project_id=c77c2e61a3db498a9e2d4fa45be9833a, user_id=2a2093be1a4f48acbacb6129749c8ed9, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:46:08.683385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8540024e-8193-4f86-900e-85337206aadf None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:46:08.697227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 438/897] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:46:08 2022] PUT /identity/v3/projects/c77c2e61a3db498a9e2d4fa45be9833a/users/2a2093be1a4f48acbacb6129749c8ed9/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:46:10.957747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4738b51-80ec-4782-81b6-35c0223c53f6 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:10.958366 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4738b51-80ec-4782-81b6-35c0223c53f6 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:10.958483 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4738b51-80ec-4782-81b6-35c0223c53f6 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:10.959973 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 460/898] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:46:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:46:10.966006 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5016ae55-f9ac-42b4-8e4f-d985b776866f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:10.966574 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5016ae55-f9ac-42b4-8e4f-d985b776866f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:10.966574 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5016ae55-f9ac-42b4-8e4f-d985b776866f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:10.985441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5016ae55-f9ac-42b4-8e4f-d985b776866f None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:46:10.986631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5016ae55-f9ac-42b4-8e4f-d985b776866f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:46:11.052245 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 439/899] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:46:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:46:11.320516 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4738b51-80ec-4782-81b6-35c0223c53f6 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:46:11.364471 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:46:11.421057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4738b51-80ec-4782-81b6-35c0223c53f6 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:46:11.455285 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3843251c-2dc1-45eb-9325-f6868b1cdcc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:46:11.455912 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3843251c-2dc1-45eb-9325-f6868b1cdcc6 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:11.456187 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3843251c-2dc1-45eb-9325-f6868b1cdcc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:11.456458 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3843251c-2dc1-45eb-9325-f6868b1cdcc6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:11.467939 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3843251c-2dc1-45eb-9325-f6868b1cdcc6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:46:11.472708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-3843251c-2dc1-45eb-9325-f6868b1cdcc6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:46:11.472708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:46:11.472708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:46:11.472708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:46:11.472708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:46:11.472708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:46:11.472708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:46:11.472708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:46:11.472708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:46:11.472708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:46:11.472708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:46:11.472708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:46:11.472708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:46:11.472708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:46:11.472708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:46:11.472708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:46:11.474056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:46:11.474056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:46:11.474056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:46:11.474056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:46:11.474056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:46:11.474056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:46:11.474056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:46:11.474056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:46:11.474056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:46:11.474056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:46:11.474056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:46:11.474056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:46:11.474056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:46:11.474056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:46:11.474056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:46:11.474056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:46:11.475156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:46:11.475156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:46:11.475156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:46:11.475156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:46:11.475156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:46:11.475156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:46:11.475156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:46:11.475156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:46:11.475156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:46:11.475156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:46:11.475156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:46:11.475156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:46:11.478985 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 461/900] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:46:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:46:11.480171 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5016ae55-f9ac-42b4-8e4f-d985b776866f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:46:11.528870 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:46:11.581298 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5016ae55-f9ac-42b4-8e4f-d985b776866f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:46:11.616441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d30898f-f0a6-4fef-84f9-6e0284dd8809 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:46:11.617045 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d30898f-f0a6-4fef-84f9-6e0284dd8809 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:11.617324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d30898f-f0a6-4fef-84f9-6e0284dd8809 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:11.617596 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d30898f-f0a6-4fef-84f9-6e0284dd8809 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:11.618255 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d30898f-f0a6-4fef-84f9-6e0284dd8809 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:46:11.618569 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d30898f-f0a6-4fef-84f9-6e0284dd8809 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:46:11.620061 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d30898f-f0a6-4fef-84f9-6e0284dd8809 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:46:11.628632 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 440/901] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:46:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:46:11.635813 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3843251c-2dc1-45eb-9325-f6868b1cdcc6 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:46:11.742209 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3843251c-2dc1-45eb-9325-f6868b1cdcc6 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:46:11.743838 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:46:11.778765 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e26749-b338-451d-8c39-6eb5864f6645 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:46:11.779437 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e26749-b338-451d-8c39-6eb5864f6645 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:11.779720 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e26749-b338-451d-8c39-6eb5864f6645 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:11.779995 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e26749-b338-451d-8c39-6eb5864f6645 None admin] PATH_INFO: `/v3/users/2a2093be1a4f48acbacb6129749c8ed9` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:11.781115 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e26749-b338-451d-8c39-6eb5864f6645 None admin] RBAC: Authorizing `identity:get_user(user_id=2a2093be1a4f48acbacb6129749c8ed9)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:46:11.782550 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e26749-b338-451d-8c39-6eb5864f6645 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:46:11.784155 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 462/902] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 16 00:46:11 2022] GET /identity/v3/users/2a2093be1a4f48acbacb6129749c8ed9 => generated 311 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:46:11.790084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d30898f-f0a6-4fef-84f9-6e0284dd8809 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:46:11.889299 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d30898f-f0a6-4fef-84f9-6e0284dd8809 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:46:11.899811 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:46:11.923578 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fb02b7-0270-4322-ae43-97dd81658657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:46:11.924183 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fb02b7-0270-4322-ae43-97dd81658657 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:11.924458 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fb02b7-0270-4322-ae43-97dd81658657 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:11.924727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fb02b7-0270-4322-ae43-97dd81658657 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:11.991834 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fb02b7-0270-4322-ae43-97dd81658657 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:46:11.993286 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fb02b7-0270-4322-ae43-97dd81658657 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:46:12.091294 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 441/903] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:46:11 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 16 00:46:12.098124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e26749-b338-451d-8c39-6eb5864f6645 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:46:12.210395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e26749-b338-451d-8c39-6eb5864f6645 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:46:12.248576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39987626-b738-47e3-b462-b08e34a27f8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:46:12.249230 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39987626-b738-47e3-b462-b08e34a27f8f None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:12.249518 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39987626-b738-47e3-b462-b08e34a27f8f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:12.249792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39987626-b738-47e3-b462-b08e34a27f8f None admin] PATH_INFO: `/v3/projects/c77c2e61a3db498a9e2d4fa45be9833a` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:12.251040 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39987626-b738-47e3-b462-b08e34a27f8f None admin] RBAC: Authorizing `identity:get_project(project_id=c77c2e61a3db498a9e2d4fa45be9833a)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:46:12.252517 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39987626-b738-47e3-b462-b08e34a27f8f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:46:12.254394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 463/904] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 16 00:46:12 2022] GET /identity/v3/projects/c77c2e61a3db498a9e2d4fa45be9833a => generated 361 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:46:12.260830 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fb02b7-0270-4322-ae43-97dd81658657 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:46:12.273623 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:46:12.366805 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fb02b7-0270-4322-ae43-97dd81658657 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:46:12.401016 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c42cf8-a34e-413f-aa58-7953ac9893d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:46:12.401630 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c42cf8-a34e-413f-aa58-7953ac9893d1 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:46:12.401914 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c42cf8-a34e-413f-aa58-7953ac9893d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:46:12.402188 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c42cf8-a34e-413f-aa58-7953ac9893d1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:46:12.402948 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c42cf8-a34e-413f-aa58-7953ac9893d1 None admin] RBAC: Adding query filter params (role.id=c5cd777d70a649028f970fd986dc484f, scope.project.id=c77c2e61a3db498a9e2d4fa45be9833a, user.id=2a2093be1a4f48acbacb6129749c8ed9) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:46:12.403261 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c42cf8-a34e-413f-aa58-7953ac9893d1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:46:12.404703 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c42cf8-a34e-413f-aa58-7953ac9893d1 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:46:12.411284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 442/905] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:46:12 2022] GET /identity/v3/role_assignments?user.id=2a2093be1a4f48acbacb6129749c8ed9&scope.project.id=c77c2e61a3db498a9e2d4fa45be9833a&role.id=c5cd777d70a649028f970fd986dc484f => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:46:22.244411 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=81 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:46:22.407961 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=63 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:11.254792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24077375-5855-4124-84b9-c7bf2bbc3e92 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:11.255443 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24077375-5855-4124-84b9-c7bf2bbc3e92 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:11.255443 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24077375-5855-4124-84b9-c7bf2bbc3e92 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:11.256876 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 464/906] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:47:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:47:11.263605 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f840d50-e47f-4f7c-8a2a-e18e7a7fe778 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:11.264470 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f840d50-e47f-4f7c-8a2a-e18e7a7fe778 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:11.264470 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f840d50-e47f-4f7c-8a2a-e18e7a7fe778 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:11.288723 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-7f840d50-e47f-4f7c-8a2a-e18e7a7fe778 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:47:11.290150 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7f840d50-e47f-4f7c-8a2a-e18e7a7fe778 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:47:11.414250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 443/907] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:47:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:47:11.728009 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24077375-5855-4124-84b9-c7bf2bbc3e92 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:11.987266 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24077375-5855-4124-84b9-c7bf2bbc3e92 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:12.062523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2a48da-fe94-4ea2-8a28-9b953c753a05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:12.070891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2a48da-fe94-4ea2-8a28-9b953c753a05 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:12.070891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2a48da-fe94-4ea2-8a28-9b953c753a05 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:12.070891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2a48da-fe94-4ea2-8a28-9b953c753a05 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:12.090451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2a48da-fe94-4ea2-8a28-9b953c753a05 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:12.093698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2a48da-fe94-4ea2-8a28-9b953c753a05 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:12.129086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-6f2a48da-fe94-4ea2-8a28-9b953c753a05 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:12.129086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:12.129086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:12.129086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:12.129086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:12.129086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:12.129086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:12.129086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:12.129086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:12.129086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:12.129086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:12.129086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:12.129086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:47:12.129086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:47:12.129086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:47:12.129086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:47:12.130439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:12.130439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:12.130439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:12.130439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:12.130439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:12.130439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:12.130439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:12.130439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:12.130439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:12.130439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:12.130439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:12.130439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:12.130439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:12.130439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:12.130439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:12.130439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:12.130439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:47:12.131613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:47:12.131613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:12.131613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:47:12.132608 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 465/908] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:47:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 406 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:12.138485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f840d50-e47f-4f7c-8a2a-e18e7a7fe778 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:12.316098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f840d50-e47f-4f7c-8a2a-e18e7a7fe778 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:12.363820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:12.373643 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d47eb62-c3e5-4308-8740-f57eb3c92ef3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:12.374557 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d47eb62-c3e5-4308-8740-f57eb3c92ef3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:12.375013 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d47eb62-c3e5-4308-8740-f57eb3c92ef3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:12.375195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d47eb62-c3e5-4308-8740-f57eb3c92ef3 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:12.376081 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d47eb62-c3e5-4308-8740-f57eb3c92ef3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:12.376439 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d47eb62-c3e5-4308-8740-f57eb3c92ef3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:12.378110 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d47eb62-c3e5-4308-8740-f57eb3c92ef3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:12.399731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 444/909] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:47:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:12.403358 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2a48da-fe94-4ea2-8a28-9b953c753a05 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:12.510087 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:12.600844 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2a48da-fe94-4ea2-8a28-9b953c753a05 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:12.682224 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbe711e-7f75-40d4-b41d-f0f1ab460730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:12.683548 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbe711e-7f75-40d4-b41d-f0f1ab460730 None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:12.684265 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbe711e-7f75-40d4-b41d-f0f1ab460730 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:12.685149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbe711e-7f75-40d4-b41d-f0f1ab460730 None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:12.686729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbe711e-7f75-40d4-b41d-f0f1ab460730 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:12.691185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbe711e-7f75-40d4-b41d-f0f1ab460730 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:12.730901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 466/910] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 16 00:47:12 2022] POST /identity/v3/users => generated 251 bytes in 324 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:47:15.432477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c595c541-68b6-4a37-97e4-569bfd08a948 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:15.433192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c595c541-68b6-4a37-97e4-569bfd08a948 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:15.433192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c595c541-68b6-4a37-97e4-569bfd08a948 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:15.434695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 445/911] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:47:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:47:15.441318 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c908c1-4391-489d-9182-69250be5e8a0 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:15.441893 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c908c1-4391-489d-9182-69250be5e8a0 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:15.441893 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c908c1-4391-489d-9182-69250be5e8a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:15.474850 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-a8c908c1-4391-489d-9182-69250be5e8a0 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:47:15.476165 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-a8c908c1-4391-489d-9182-69250be5e8a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:47:15.590146 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 467/912] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:47:15 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 16 00:47:15.905646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c595c541-68b6-4a37-97e4-569bfd08a948 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:16.082699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c595c541-68b6-4a37-97e4-569bfd08a948 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:16.149803 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303e1108-060b-48cf-afab-879f9e2aaf31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:16.151712 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303e1108-060b-48cf-afab-879f9e2aaf31 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:16.152777 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303e1108-060b-48cf-afab-879f9e2aaf31 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:16.153633 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303e1108-060b-48cf-afab-879f9e2aaf31 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:16.179729 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303e1108-060b-48cf-afab-879f9e2aaf31 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:47:16.188671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-303e1108-060b-48cf-afab-879f9e2aaf31 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:47:16.188671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:16.188671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:16.188671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:16.188671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:16.188671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:16.188671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:16.188671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:16.188671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:16.188671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:16.188671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:16.188671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:16.188671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:47:16.188671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:47:16.188671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:47:16.188671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:47:16.191117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:47:16.191117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:47:16.191117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:47:16.191117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:47:16.191117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:16.191117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:16.191117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:16.191117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:16.191117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:16.191117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:16.191117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:16.191117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:16.191117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:16.191117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:16.191117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:16.191117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:16.193106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:16.193106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:16.193106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:16.193106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:16.193106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:47:16.193106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:47:16.193106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:47:16.193106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:47:16.193106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:47:16.193106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:47:16.193106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:47:16.193106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:16.206951 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 446/913] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 16 00:47:15 2022] GET /identity/v3/roles/service => generated 85 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:16.207302 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c908c1-4391-489d-9182-69250be5e8a0 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:16.295849 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:16.445937 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c908c1-4391-489d-9182-69250be5e8a0 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:16.505384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4bd9ec-3200-4b60-889e-481497f425e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:16.506728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4bd9ec-3200-4b60-889e-481497f425e3 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:16.507019 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4bd9ec-3200-4b60-889e-481497f425e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:16.507578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4bd9ec-3200-4b60-889e-481497f425e3 None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:16.508659 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4bd9ec-3200-4b60-889e-481497f425e3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:16.509254 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4bd9ec-3200-4b60-889e-481497f425e3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:16.511429 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4bd9ec-3200-4b60-889e-481497f425e3 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:16.525039 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 468/914] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 16 00:47:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:16.534396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303e1108-060b-48cf-afab-879f9e2aaf31 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:16.606852 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:16.691596 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303e1108-060b-48cf-afab-879f9e2aaf31 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:16.751687 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c97fc6e-3364-41fc-b64e-7b04886e9069 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:16.753136 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c97fc6e-3364-41fc-b64e-7b04886e9069 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:16.753816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c97fc6e-3364-41fc-b64e-7b04886e9069 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:16.754207 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c97fc6e-3364-41fc-b64e-7b04886e9069 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:16.775085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c97fc6e-3364-41fc-b64e-7b04886e9069 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:16.778170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c97fc6e-3364-41fc-b64e-7b04886e9069 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:16.805941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-2c97fc6e-3364-41fc-b64e-7b04886e9069 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:16.805941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:16.805941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:16.805941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:16.805941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:16.805941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:16.805941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:16.805941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:16.805941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:16.805941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:16.805941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:16.805941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:16.805941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:47:16.805941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:47:16.805941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:47:16.805941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:47:16.807108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:16.807108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:16.807108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:16.807108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:16.807108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:16.807108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:16.807108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:16.807108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:16.807108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:16.807108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:16.807108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:16.807108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:16.807108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:16.807108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:16.807108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:16.807108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:16.807108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:47:16.808039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:47:16.808039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:16.808039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:16.818002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 447/915] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:47:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:16.819412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4bd9ec-3200-4b60-889e-481497f425e3 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:17.045551 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:17.053760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4bd9ec-3200-4b60-889e-481497f425e3 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:17.122921 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94aed473-cf29-444a-bf9e-777e36b3a46d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:17.126743 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94aed473-cf29-444a-bf9e-777e36b3a46d None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:17.127216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94aed473-cf29-444a-bf9e-777e36b3a46d None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:17.127665 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94aed473-cf29-444a-bf9e-777e36b3a46d None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:17.128841 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94aed473-cf29-444a-bf9e-777e36b3a46d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:17.129382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94aed473-cf29-444a-bf9e-777e36b3a46d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:17.132662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94aed473-cf29-444a-bf9e-777e36b3a46d None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:17.152553 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 469/916] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:47:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 335 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:17.160140 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c97fc6e-3364-41fc-b64e-7b04886e9069 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:17.366126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c97fc6e-3364-41fc-b64e-7b04886e9069 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:17.369496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:17.437071 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fd214f-9726-4557-bce5-a5d18525a5d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:17.438151 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fd214f-9726-4557-bce5-a5d18525a5d3 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:17.438646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fd214f-9726-4557-bce5-a5d18525a5d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:17.439182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fd214f-9726-4557-bce5-a5d18525a5d3 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:17.456279 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fd214f-9726-4557-bce5-a5d18525a5d3 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:17.458148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fd214f-9726-4557-bce5-a5d18525a5d3 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:17.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-62fd214f-9726-4557-bce5-a5d18525a5d3 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 16 00:47:17.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:17.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:17.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:17.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:17.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:17.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:17.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:17.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:17.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:17.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:17.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:17.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:47:17.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:47:17.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:47:17.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:17.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:17.483586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:17.483586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:17.483586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:47:17.483586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:47:17.483586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:47:17.483586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:47:17.483586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:47:17.483586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:47:17.483586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 16 00:47:17.483586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:17.483586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 448/917] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:47:17 2022] GET /identity/v3/users/placement => generated 87 bytes in 322 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:17.489495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94aed473-cf29-444a-bf9e-777e36b3a46d None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:17.683012 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94aed473-cf29-444a-bf9e-777e36b3a46d None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:17.736678 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e6d2b3-d87a-44c5-a012-b5a1fd7bf7b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:17.737860 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e6d2b3-d87a-44c5-a012-b5a1fd7bf7b4 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:17.738706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e6d2b3-d87a-44c5-a012-b5a1fd7bf7b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:17.739553 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e6d2b3-d87a-44c5-a012-b5a1fd7bf7b4 None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:17.741431 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e6d2b3-d87a-44c5-a012-b5a1fd7bf7b4 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:17.742096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e6d2b3-d87a-44c5-a012-b5a1fd7bf7b4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:17.745359 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e6d2b3-d87a-44c5-a012-b5a1fd7bf7b4 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:17.782798 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 470/918] 10.222.0.25 () {56 vars in 1143 bytes} [Sun Oct 16 00:47:17 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:47:17.784975 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fd214f-9726-4557-bce5-a5d18525a5d3 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:17.970275 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fd214f-9726-4557-bce5-a5d18525a5d3 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:18.029201 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0878aec4-8161-43e7-b426-392db55b7c94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:18.030240 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0878aec4-8161-43e7-b426-392db55b7c94 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:18.030724 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0878aec4-8161-43e7-b426-392db55b7c94 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:18.031165 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0878aec4-8161-43e7-b426-392db55b7c94 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:18.053075 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0878aec4-8161-43e7-b426-392db55b7c94 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:18.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0878aec4-8161-43e7-b426-392db55b7c94 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:18.091233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-0878aec4-8161-43e7-b426-392db55b7c94 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:18.091233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:18.091233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:18.091233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:18.091233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:18.091233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:18.091233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:18.091233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:18.091233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:18.091233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:18.091233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:18.091233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:18.091233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:47:18.091233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:47:18.091233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:47:18.091233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:47:18.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:18.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:18.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:18.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:18.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:18.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:18.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:18.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:18.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:18.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:18.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:18.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:18.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:18.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:18.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:18.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:18.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:47:18.093497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:47:18.093497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:18.093497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:18.093675 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 449/919] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:47:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:18.104843 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e6d2b3-d87a-44c5-a012-b5a1fd7bf7b4 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:18.139718 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:18.261801 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e6d2b3-d87a-44c5-a012-b5a1fd7bf7b4 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:18.309413 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e978336-05dc-47f1-9fd1-64bf5cb3a6b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:18.310168 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e978336-05dc-47f1-9fd1-64bf5cb3a6b2 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:18.310451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e978336-05dc-47f1-9fd1-64bf5cb3a6b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:18.310725 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e978336-05dc-47f1-9fd1-64bf5cb3a6b2 None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:18.311448 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e978336-05dc-47f1-9fd1-64bf5cb3a6b2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:18.311771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e978336-05dc-47f1-9fd1-64bf5cb3a6b2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:18.313702 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e978336-05dc-47f1-9fd1-64bf5cb3a6b2 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:18.327845 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 471/920] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:47:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:47:18.336165 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0878aec4-8161-43e7-b426-392db55b7c94 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:18.357891 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:18.500127 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0878aec4-8161-43e7-b426-392db55b7c94 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:18.545460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343f4507-d442-4654-a8f8-bfc9c5dc4cf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:18.546207 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343f4507-d442-4654-a8f8-bfc9c5dc4cf2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:18.546555 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343f4507-d442-4654-a8f8-bfc9c5dc4cf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:18.547028 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343f4507-d442-4654-a8f8-bfc9c5dc4cf2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:18.556683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343f4507-d442-4654-a8f8-bfc9c5dc4cf2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:18.558944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343f4507-d442-4654-a8f8-bfc9c5dc4cf2 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:18.572938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-343f4507-d442-4654-a8f8-bfc9c5dc4cf2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:47:18.572938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:18.572938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:18.572938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:18.572938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:18.572938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:18.572938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:18.572938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:18.572938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:18.572938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:18.572938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:18.572938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:18.572938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:47:18.572938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:47:18.572938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:47:18.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:47:18.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:18.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:18.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:18.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:18.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:18.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:18.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:18.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:18.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:18.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:18.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:18.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:18.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:18.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:18.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:18.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:18.575348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:47:18.575348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:47:18.575348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:47:18.575348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:47:18.575348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:47:18.575348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:47:18.575348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:47:18.575348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:18.575348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 450/921] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:47:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:18.580484 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e978336-05dc-47f1-9fd1-64bf5cb3a6b2 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:18.693652 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:18.724845 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e978336-05dc-47f1-9fd1-64bf5cb3a6b2 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:18.774913 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceeca910-3358-4224-b3b1-1d163b6f3b6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:18.775937 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeca910-3358-4224-b3b1-1d163b6f3b6c None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:18.776261 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeca910-3358-4224-b3b1-1d163b6f3b6c None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:18.776589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeca910-3358-4224-b3b1-1d163b6f3b6c None admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:18.777637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceeca910-3358-4224-b3b1-1d163b6f3b6c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:18.778086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceeca910-3358-4224-b3b1-1d163b6f3b6c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:18.780003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceeca910-3358-4224-b3b1-1d163b6f3b6c None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:18.804033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 472/922] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:47:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:18.805673 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343f4507-d442-4654-a8f8-bfc9c5dc4cf2 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:18.930738 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:18.981606 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343f4507-d442-4654-a8f8-bfc9c5dc4cf2 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:19.026476 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbcd8ed-e2e9-43cb-997d-92bb5f496ec0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:19.027210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbcd8ed-e2e9-43cb-997d-92bb5f496ec0 None admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:19.027506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbcd8ed-e2e9-43cb-997d-92bb5f496ec0 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:19.027975 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbcd8ed-e2e9-43cb-997d-92bb5f496ec0 None admin] PATH_INFO: `/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/87785ccd37a642e08a0ad34a4fb1c754/roles/bfe57c41a4da4b25b9b3e4761f220a46` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:19.046902 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbcd8ed-e2e9-43cb-997d-92bb5f496ec0 None admin] RBAC: Authorizing `identity:create_grant(project_id=7deb2c3dae214df7895b40e7339bd6ea, user_id=87785ccd37a642e08a0ad34a4fb1c754, role_id=bfe57c41a4da4b25b9b3e4761f220a46)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:19.049163 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbcd8ed-e2e9-43cb-997d-92bb5f496ec0 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:19.065169 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 451/923] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:47:18 2022] PUT /identity/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/87785ccd37a642e08a0ad34a4fb1c754/roles/bfe57c41a4da4b25b9b3e4761f220a46 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:47:21.833141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4b607f-181b-4350-8b94-52f2c368eab9 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:21.833141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4b607f-181b-4350-8b94-52f2c368eab9 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:21.834605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4b607f-181b-4350-8b94-52f2c368eab9 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:21.838437 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 473/924] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:47:21 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:47:21.845641 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7ec3f7-5ecc-4e67-a147-82c483a2c15d None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:21.846170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7ec3f7-5ecc-4e67-a147-82c483a2c15d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:21.846303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7ec3f7-5ecc-4e67-a147-82c483a2c15d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:21.866027 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-bb7ec3f7-5ecc-4e67-a147-82c483a2c15d None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:47:21.867291 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bb7ec3f7-5ecc-4e67-a147-82c483a2c15d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:47:21.958488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 452/925] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:47:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:47:22.282574 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4b607f-181b-4350-8b94-52f2c368eab9 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:22.427900 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4b607f-181b-4350-8b94-52f2c368eab9 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:22.480828 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1618d4-230f-4f18-8b0f-36cc97fe2143 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:22.481475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1618d4-230f-4f18-8b0f-36cc97fe2143 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:22.481757 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1618d4-230f-4f18-8b0f-36cc97fe2143 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:22.482071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1618d4-230f-4f18-8b0f-36cc97fe2143 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:22.492104 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:22.497454 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1618d4-230f-4f18-8b0f-36cc97fe2143 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:47:22.509512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-ed1618d4-230f-4f18-8b0f-36cc97fe2143 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:47:22.509512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:22.509512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:22.509512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:22.509512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:22.509512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:22.509512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:22.509512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:22.509512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:22.509512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:22.509512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:22.509512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:22.509512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:47:22.509512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:47:22.509512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:47:22.509512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:47:22.510824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:47:22.510824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:47:22.510824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:47:22.510824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:47:22.510824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:22.510824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:22.510824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:22.510824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:22.510824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:22.510824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:22.510824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:22.510824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:22.510824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:22.510824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:22.510824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:22.510824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:22.512091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:22.512091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:22.512091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:22.512091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:22.512091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:47:22.512091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:47:22.512091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:47:22.512091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:47:22.512091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:47:22.512091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:47:22.512091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 16 00:47:22.512091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:47:22.512091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 474/926] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 16 00:47:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:47:22.514782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7ec3f7-5ecc-4e67-a147-82c483a2c15d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:22.592853 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:22.651632 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7ec3f7-5ecc-4e67-a147-82c483a2c15d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:22.702089 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76c1181-1b22-40c3-b0c5-00eb2c8a7ad8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:22.702944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76c1181-1b22-40c3-b0c5-00eb2c8a7ad8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:22.703615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76c1181-1b22-40c3-b0c5-00eb2c8a7ad8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:22.704020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76c1181-1b22-40c3-b0c5-00eb2c8a7ad8 None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:22.705096 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76c1181-1b22-40c3-b0c5-00eb2c8a7ad8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:22.705641 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76c1181-1b22-40c3-b0c5-00eb2c8a7ad8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:22.707796 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76c1181-1b22-40c3-b0c5-00eb2c8a7ad8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:22.723208 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 453/927] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 16 00:47:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:22.730164 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1618d4-230f-4f18-8b0f-36cc97fe2143 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:22.863502 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1618d4-230f-4f18-8b0f-36cc97fe2143 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:22.905465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cdce26-d457-4365-99d0-3bfc6a5ca4fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:22.906133 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cdce26-d457-4365-99d0-3bfc6a5ca4fa None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:22.907287 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cdce26-d457-4365-99d0-3bfc6a5ca4fa None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:22.907287 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cdce26-d457-4365-99d0-3bfc6a5ca4fa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:22.918445 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cdce26-d457-4365-99d0-3bfc6a5ca4fa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:22.920038 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cdce26-d457-4365-99d0-3bfc6a5ca4fa None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:22.941856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-e0cdce26-d457-4365-99d0-3bfc6a5ca4fa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:22.941856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:22.941856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:22.941856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:22.941856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:22.941856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:22.941856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:22.941856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:22.941856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:22.941856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:22.941856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:22.941856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:22.941856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:47:22.941856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:47:22.941856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:47:22.941856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:47:22.943157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:22.943157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:22.943157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:22.943157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:22.943157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:22.943157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:22.943157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:22.943157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:22.943157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:22.943157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:22.943157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:22.943157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:22.943157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:22.943157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:22.943157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:22.943157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:22.943157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:47:22.944370 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:47:22.944370 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:22.944370 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:47:22.944370 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 475/928] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:47:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:47:22.949261 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76c1181-1b22-40c3-b0c5-00eb2c8a7ad8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:23.082290 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76c1181-1b22-40c3-b0c5-00eb2c8a7ad8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:23.100880 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:23.124580 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f2c44f-a336-4398-b531-af31eee7a808 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:23.125114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f2c44f-a336-4398-b531-af31eee7a808 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:23.125732 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f2c44f-a336-4398-b531-af31eee7a808 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:23.126002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f2c44f-a336-4398-b531-af31eee7a808 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:23.126655 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f2c44f-a336-4398-b531-af31eee7a808 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:23.126990 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f2c44f-a336-4398-b531-af31eee7a808 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:23.128372 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f2c44f-a336-4398-b531-af31eee7a808 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:23.142105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 454/929] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:47:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:23.148612 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cdce26-d457-4365-99d0-3bfc6a5ca4fa None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:23.223779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:23.288659 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cdce26-d457-4365-99d0-3bfc6a5ca4fa None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:23.325286 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1331dc-1f17-471c-9d0c-088571ba292c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:23.325965 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1331dc-1f17-471c-9d0c-088571ba292c None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:23.326266 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1331dc-1f17-471c-9d0c-088571ba292c None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:23.326527 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1331dc-1f17-471c-9d0c-088571ba292c None admin] PATH_INFO: `/v3/users/placement` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:23.338956 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1331dc-1f17-471c-9d0c-088571ba292c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:23.338956 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1331dc-1f17-471c-9d0c-088571ba292c None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:23.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-cd1331dc-1f17-471c-9d0c-088571ba292c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 16 00:47:23.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:23.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:23.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:23.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:23.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:23.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:23.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:23.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:23.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:23.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:23.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:23.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:47:23.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:47:23.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:47:23.353073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:23.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:23.355557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:23.355557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:23.355557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:47:23.355557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:47:23.355557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:47:23.355557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:47:23.355557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:47:23.355557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:47:23.355557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 16 00:47:23.355557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:47:23.355557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 476/930] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:47:23 2022] GET /identity/v3/users/placement => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:23.360821 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f2c44f-a336-4398-b531-af31eee7a808 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:23.498686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f2c44f-a336-4398-b531-af31eee7a808 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:23.555675 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a181985c-a7f5-4b4c-a6be-28f585e5b2c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:23.556313 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a181985c-a7f5-4b4c-a6be-28f585e5b2c5 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:23.556597 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a181985c-a7f5-4b4c-a6be-28f585e5b2c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:23.556872 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a181985c-a7f5-4b4c-a6be-28f585e5b2c5 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:23.557625 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a181985c-a7f5-4b4c-a6be-28f585e5b2c5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:23.557931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a181985c-a7f5-4b4c-a6be-28f585e5b2c5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:23.559440 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a181985c-a7f5-4b4c-a6be-28f585e5b2c5 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:23.580295 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 455/931] 10.222.0.25 () {56 vars in 1143 bytes} [Sun Oct 16 00:47:23 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:47:23.587507 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1331dc-1f17-471c-9d0c-088571ba292c None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:23.733039 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1331dc-1f17-471c-9d0c-088571ba292c None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:23.747156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:23.778049 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879eeceb-9a31-41ce-a0ac-65b5e8458316 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:23.779185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879eeceb-9a31-41ce-a0ac-65b5e8458316 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:23.779538 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879eeceb-9a31-41ce-a0ac-65b5e8458316 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:23.779915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879eeceb-9a31-41ce-a0ac-65b5e8458316 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:23.928704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879eeceb-9a31-41ce-a0ac-65b5e8458316 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:23.930285 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879eeceb-9a31-41ce-a0ac-65b5e8458316 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:24.067860 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 477/932] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:47:23 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 475 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 16 00:47:24.069367 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a181985c-a7f5-4b4c-a6be-28f585e5b2c5 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:24.115736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:24.213222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a181985c-a7f5-4b4c-a6be-28f585e5b2c5 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:24.257331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdcdc2b-e466-4b66-b095-3d0c35af9920 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:24.258160 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdcdc2b-e466-4b66-b095-3d0c35af9920 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:24.258401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdcdc2b-e466-4b66-b095-3d0c35af9920 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:24.258730 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdcdc2b-e466-4b66-b095-3d0c35af9920 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:24.276586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdcdc2b-e466-4b66-b095-3d0c35af9920 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:24.278549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdcdc2b-e466-4b66-b095-3d0c35af9920 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:24.301228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-9fdcdc2b-e466-4b66-b095-3d0c35af9920 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:24.301228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:24.301228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:24.301228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:24.301228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:24.301228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:24.301228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:24.301228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:24.301228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:24.301228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:24.301228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:24.301228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:24.301228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:47:24.301228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:47:24.301228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:47:24.301228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:47:24.303340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:24.303340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:24.303340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:24.303340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:24.303340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:24.303340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:24.303340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:24.303340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:24.303340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:24.303340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:24.303340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:24.303340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:24.303340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:24.303340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:24.303340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:24.303340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:24.303340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:47:24.304839 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:47:24.304839 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:24.304839 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:24.304839 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 456/933] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:47:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:24.310459 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879eeceb-9a31-41ce-a0ac-65b5e8458316 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:24.326125 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:24.444288 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879eeceb-9a31-41ce-a0ac-65b5e8458316 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:24.495425 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55308bdf-0a17-437a-ae6a-fc803c82a0cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:24.496097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55308bdf-0a17-437a-ae6a-fc803c82a0cc None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:24.496426 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55308bdf-0a17-437a-ae6a-fc803c82a0cc None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:24.496691 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55308bdf-0a17-437a-ae6a-fc803c82a0cc None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:24.497408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55308bdf-0a17-437a-ae6a-fc803c82a0cc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:24.497718 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55308bdf-0a17-437a-ae6a-fc803c82a0cc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:24.499149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55308bdf-0a17-437a-ae6a-fc803c82a0cc None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:24.512858 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 478/934] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:47:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:24.520527 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdcdc2b-e466-4b66-b095-3d0c35af9920 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:24.633130 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:24.651381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdcdc2b-e466-4b66-b095-3d0c35af9920 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:24.701474 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd557c7-a013-4797-ab5d-ae7d635da4b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:24.702282 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd557c7-a013-4797-ab5d-ae7d635da4b2 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:24.702574 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd557c7-a013-4797-ab5d-ae7d635da4b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:24.702893 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd557c7-a013-4797-ab5d-ae7d635da4b2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:24.721818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd557c7-a013-4797-ab5d-ae7d635da4b2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:24.723562 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd557c7-a013-4797-ab5d-ae7d635da4b2 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:24.738200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-fcd557c7-a013-4797-ab5d-ae7d635da4b2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:47:24.738200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:24.738200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:24.738200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:24.738200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:24.738200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:24.738200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:24.738200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:24.738200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:24.738200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:24.738200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:24.738200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:24.738200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:47:24.738200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:47:24.738200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:47:24.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:47:24.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:24.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:24.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:24.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:24.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:24.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:24.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:24.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:24.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:24.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:24.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:24.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:24.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:24.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:24.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:24.747088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:24.749105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:47:24.749105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:47:24.749105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:47:24.749105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:47:24.749105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:47:24.749105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:47:24.749105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:47:24.749105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:24.749105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 457/935] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:47:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:24.751503 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55308bdf-0a17-437a-ae6a-fc803c82a0cc None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:24.865086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:24.905328 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55308bdf-0a17-437a-ae6a-fc803c82a0cc None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:24.948300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681bab14-2675-4928-89ef-3a47222eb086 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:24.955289 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681bab14-2675-4928-89ef-3a47222eb086 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:24.955720 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681bab14-2675-4928-89ef-3a47222eb086 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:24.956099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681bab14-2675-4928-89ef-3a47222eb086 None admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:24.957366 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681bab14-2675-4928-89ef-3a47222eb086 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:24.957665 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681bab14-2675-4928-89ef-3a47222eb086 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:24.959383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681bab14-2675-4928-89ef-3a47222eb086 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:24.975954 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 479/936] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:47:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:24.983611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd557c7-a013-4797-ab5d-ae7d635da4b2 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:25.112437 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd557c7-a013-4797-ab5d-ae7d635da4b2 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:25.156942 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405ccf50-a0d8-48d0-b147-65d21c1ee97a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:25.159198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405ccf50-a0d8-48d0-b147-65d21c1ee97a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:25.159477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405ccf50-a0d8-48d0-b147-65d21c1ee97a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:25.159756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405ccf50-a0d8-48d0-b147-65d21c1ee97a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:25.160501 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405ccf50-a0d8-48d0-b147-65d21c1ee97a None admin] RBAC: Adding query filter params (role.id=bfe57c41a4da4b25b9b3e4761f220a46, scope.project.id=7deb2c3dae214df7895b40e7339bd6ea, user.id=87785ccd37a642e08a0ad34a4fb1c754) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:25.160810 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405ccf50-a0d8-48d0-b147-65d21c1ee97a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:25.162264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405ccf50-a0d8-48d0-b147-65d21c1ee97a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:25.169884 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 458/937] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:47:24 2022] GET /identity/v3/role_assignments?user.id=87785ccd37a642e08a0ad34a4fb1c754&scope.project.id=7deb2c3dae214df7895b40e7339bd6ea&role.id=bfe57c41a4da4b25b9b3e4761f220a46 => generated 608 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:47:27.502531 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb362a6-60f0-4fef-87c1-44ae405f6d05 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:27.503104 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb362a6-60f0-4fef-87c1-44ae405f6d05 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:27.503104 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb362a6-60f0-4fef-87c1-44ae405f6d05 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:27.504516 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 480/938] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:47:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:47:27.511550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d3da21-bf35-4cc2-a992-c2d4c21cc547 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:27.511550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d3da21-bf35-4cc2-a992-c2d4c21cc547 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:27.511550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d3da21-bf35-4cc2-a992-c2d4c21cc547 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:27.531902 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f4d3da21-bf35-4cc2-a992-c2d4c21cc547 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:47:27.533425 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f4d3da21-bf35-4cc2-a992-c2d4c21cc547 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:47:27.536909 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:27.647434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 459/939] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:47:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:47:28.026968 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb362a6-60f0-4fef-87c1-44ae405f6d05 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:28.180480 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb362a6-60f0-4fef-87c1-44ae405f6d05 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:28.220250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:28.227148 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b62f96d-892e-4bb4-b02e-0579c3c6eb21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:28.227893 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b62f96d-892e-4bb4-b02e-0579c3c6eb21 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:28.228180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b62f96d-892e-4bb4-b02e-0579c3c6eb21 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:28.228458 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b62f96d-892e-4bb4-b02e-0579c3c6eb21 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:28.240697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b62f96d-892e-4bb4-b02e-0579c3c6eb21 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:47:28.245635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-1b62f96d-892e-4bb4-b02e-0579c3c6eb21 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:47:28.245635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:28.245635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:28.245635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:28.245635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:28.245635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:28.245635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:28.245635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:28.245635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:28.245635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:28.245635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:28.245635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:28.245635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:47:28.245635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:47:28.245635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:47:28.245635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:47:28.247361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:47:28.247361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:47:28.247361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:47:28.247361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:47:28.247361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:28.247361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:28.247361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:28.247361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:28.247361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:28.247361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:28.247361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:28.247361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:28.247361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:28.247361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:28.247361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:28.247361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:28.248692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:28.248692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:28.248692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:28.248692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:28.248692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:47:28.248692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:47:28.248692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:47:28.248692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:47:28.248692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:47:28.248692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:47:28.248692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:47:28.248692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:47:28.248692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 481/940] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:47:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:47:28.254063 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d3da21-bf35-4cc2-a992-c2d4c21cc547 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:28.394171 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d3da21-bf35-4cc2-a992-c2d4c21cc547 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:28.443374 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d449c3bf-9775-4c2c-b5ce-eed16d8edf0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:28.444172 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d449c3bf-9775-4c2c-b5ce-eed16d8edf0a None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:28.444459 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d449c3bf-9775-4c2c-b5ce-eed16d8edf0a None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:28.444732 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d449c3bf-9775-4c2c-b5ce-eed16d8edf0a None admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:28.445408 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d449c3bf-9775-4c2c-b5ce-eed16d8edf0a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:28.445724 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d449c3bf-9775-4c2c-b5ce-eed16d8edf0a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:28.447369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d449c3bf-9775-4c2c-b5ce-eed16d8edf0a None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:28.458244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 460/941] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:47:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:28.464674 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b62f96d-892e-4bb4-b02e-0579c3c6eb21 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:28.595306 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b62f96d-892e-4bb4-b02e-0579c3c6eb21 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:28.653234 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba36db0-2b86-4670-b62a-0ae48def5662 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:28.654014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba36db0-2b86-4670-b62a-0ae48def5662 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:28.654417 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba36db0-2b86-4670-b62a-0ae48def5662 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:28.654809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba36db0-2b86-4670-b62a-0ae48def5662 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:28.671222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba36db0-2b86-4670-b62a-0ae48def5662 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:28.673775 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba36db0-2b86-4670-b62a-0ae48def5662 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:28.690298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-3ba36db0-2b86-4670-b62a-0ae48def5662 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:28.690298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:28.690298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:28.690298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:28.690298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:28.690298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:28.690298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:28.690298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:28.690298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:28.690298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:28.690298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:28.690298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:28.690298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:47:28.690298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:47:28.690298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:47:28.690298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:47:28.692046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:28.692046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:28.692046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:28.692046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:28.692046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:28.692046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:28.692046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:28.692046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:28.692046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:28.692046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:28.692046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:28.692046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:28.692046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:28.692046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:28.692046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:28.692046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:28.692046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:47:28.693525 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:47:28.693525 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:28.693525 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:47:28.693525 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 482/942] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:47:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 16 00:47:28.698143 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d449c3bf-9775-4c2c-b5ce-eed16d8edf0a None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:28.702782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:28.829742 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d449c3bf-9775-4c2c-b5ce-eed16d8edf0a None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:28.878123 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48802677-03fa-4215-86dd-23fd050dc923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:28.878777 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48802677-03fa-4215-86dd-23fd050dc923 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:28.879094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48802677-03fa-4215-86dd-23fd050dc923 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:28.879375 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48802677-03fa-4215-86dd-23fd050dc923 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:28.880299 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48802677-03fa-4215-86dd-23fd050dc923 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:28.880299 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48802677-03fa-4215-86dd-23fd050dc923 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:28.881722 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48802677-03fa-4215-86dd-23fd050dc923 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:28.897868 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 461/943] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:47:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:47:28.904047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba36db0-2b86-4670-b62a-0ae48def5662 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:28.957982 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:29.029833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba36db0-2b86-4670-b62a-0ae48def5662 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:29.068808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7dce673-d4cc-48ee-8a37-0edec429ab54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:29.069452 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dce673-d4cc-48ee-8a37-0edec429ab54 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:29.069737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dce673-d4cc-48ee-8a37-0edec429ab54 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:29.070011 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dce673-d4cc-48ee-8a37-0edec429ab54 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:29.077755 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7dce673-d4cc-48ee-8a37-0edec429ab54 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:29.079555 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7dce673-d4cc-48ee-8a37-0edec429ab54 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:29.092630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-c7dce673-d4cc-48ee-8a37-0edec429ab54 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 16 00:47:29.092630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:29.092630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:29.092630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:29.092630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:29.092630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:29.092630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:29.092630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:29.092630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:29.092630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:29.092630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:29.092630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:29.092630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:47:29.092630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:47:29.092630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:47:29.092630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:29.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:29.096465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:29.096465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:29.096465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:47:29.096465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:47:29.096465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:47:29.096465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:47:29.096465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:47:29.096465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:47:29.096465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 16 00:47:29.096465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:47:29.096465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 483/944] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:47:28 2022] GET /identity/v3/users/placement => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:29.101114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48802677-03fa-4215-86dd-23fd050dc923 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:29.181705 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:29.233515 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48802677-03fa-4215-86dd-23fd050dc923 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:29.283151 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01904cc6-55ea-4882-94ca-98698f9e0645 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:29.283786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01904cc6-55ea-4882-94ca-98698f9e0645 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:29.284152 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01904cc6-55ea-4882-94ca-98698f9e0645 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:29.284506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01904cc6-55ea-4882-94ca-98698f9e0645 None admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:29.285450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01904cc6-55ea-4882-94ca-98698f9e0645 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:29.285756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01904cc6-55ea-4882-94ca-98698f9e0645 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:29.287625 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01904cc6-55ea-4882-94ca-98698f9e0645 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:29.314975 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 462/945] 10.222.0.25 () {56 vars in 1143 bytes} [Sun Oct 16 00:47:29 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:29.321334 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7dce673-d4cc-48ee-8a37-0edec429ab54 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:29.458843 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7dce673-d4cc-48ee-8a37-0edec429ab54 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:29.460650 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:29.510407 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0aeb54-3096-42a6-ab53-b3a1ed4c09b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:29.511113 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0aeb54-3096-42a6-ab53-b3a1ed4c09b5 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:29.511399 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0aeb54-3096-42a6-ab53-b3a1ed4c09b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:29.511669 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0aeb54-3096-42a6-ab53-b3a1ed4c09b5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:29.529192 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0aeb54-3096-42a6-ab53-b3a1ed4c09b5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:29.532634 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0aeb54-3096-42a6-ab53-b3a1ed4c09b5 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:29.553698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-ad0aeb54-3096-42a6-ab53-b3a1ed4c09b5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:29.553698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:29.553698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:29.553698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:29.553698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:29.553698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:29.553698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:29.553698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:29.553698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:29.553698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:29.553698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:29.553698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:29.553698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:47:29.553698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:47:29.553698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:47:29.553698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:47:29.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:29.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:29.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:29.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:29.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:29.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:29.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:29.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:29.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:29.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:29.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:29.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:29.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:29.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:29.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:29.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:29.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:47:29.559580 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:47:29.559580 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:29.559580 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:47:29.559580 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 484/946] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:47:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:29.562813 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01904cc6-55ea-4882-94ca-98698f9e0645 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:29.722133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01904cc6-55ea-4882-94ca-98698f9e0645 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:29.749192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:29.782105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f55267-8b97-4d4a-b8f0-dd73cd780e39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:29.783189 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f55267-8b97-4d4a-b8f0-dd73cd780e39 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:29.783842 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f55267-8b97-4d4a-b8f0-dd73cd780e39 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:29.784452 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f55267-8b97-4d4a-b8f0-dd73cd780e39 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:29.785785 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f55267-8b97-4d4a-b8f0-dd73cd780e39 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:29.786255 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f55267-8b97-4d4a-b8f0-dd73cd780e39 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:29.789307 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f55267-8b97-4d4a-b8f0-dd73cd780e39 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:29.810713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 463/947] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:47:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:29.817454 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0aeb54-3096-42a6-ab53-b3a1ed4c09b5 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:29.941844 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0aeb54-3096-42a6-ab53-b3a1ed4c09b5 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:29.983301 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7d81fd-40d1-4860-8c1a-60fa2795e5a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:29.984066 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7d81fd-40d1-4860-8c1a-60fa2795e5a4 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:29.984570 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7d81fd-40d1-4860-8c1a-60fa2795e5a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:29.984893 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7d81fd-40d1-4860-8c1a-60fa2795e5a4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:29.995793 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7d81fd-40d1-4860-8c1a-60fa2795e5a4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:29.997679 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7d81fd-40d1-4860-8c1a-60fa2795e5a4 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:30.013693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-8f7d81fd-40d1-4860-8c1a-60fa2795e5a4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:47:30.013693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:30.013693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:30.013693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:30.013693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:30.013693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:30.013693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:30.013693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:30.013693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:30.013693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:30.013693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:30.013693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:30.013693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:47:30.013693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:47:30.013693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:47:30.014779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:47:30.014779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:30.014779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:30.014779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:30.014779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:30.014779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:30.014779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:30.014779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:30.014779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:30.014779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:30.014779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:30.014779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:30.014779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:30.014779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:30.014779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:30.014779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:30.014779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:30.015918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:47:30.015918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:47:30.015918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:47:30.015918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:47:30.015918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:47:30.015918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:47:30.015918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:47:30.015918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:47:30.015918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 485/948] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:47:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:30.021707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f55267-8b97-4d4a-b8f0-dd73cd780e39 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:30.158683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f55267-8b97-4d4a-b8f0-dd73cd780e39 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:30.203624 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b450501e-a59e-4f3a-a555-b94f8bade934 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:30.204652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b450501e-a59e-4f3a-a555-b94f8bade934 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:30.204652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b450501e-a59e-4f3a-a555-b94f8bade934 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:30.205070 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b450501e-a59e-4f3a-a555-b94f8bade934 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:30.206107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b450501e-a59e-4f3a-a555-b94f8bade934 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:30.206536 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b450501e-a59e-4f3a-a555-b94f8bade934 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:30.208235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b450501e-a59e-4f3a-a555-b94f8bade934 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:30.225775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 464/949] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:47:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:30.232971 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7d81fd-40d1-4860-8c1a-60fa2795e5a4 None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:30.255931 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:30.363585 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7d81fd-40d1-4860-8c1a-60fa2795e5a4 None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:30.416880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391fb887-843d-4b32-aa86-32695bb3c1a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:30.417806 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391fb887-843d-4b32-aa86-32695bb3c1a3 None admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:30.418302 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391fb887-843d-4b32-aa86-32695bb3c1a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:30.418722 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391fb887-843d-4b32-aa86-32695bb3c1a3 None admin] PATH_INFO: `/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/87785ccd37a642e08a0ad34a4fb1c754/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:30.421454 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391fb887-843d-4b32-aa86-32695bb3c1a3 None admin] RBAC: Authorizing `identity:create_grant(project_id=7deb2c3dae214df7895b40e7339bd6ea, user_id=87785ccd37a642e08a0ad34a4fb1c754, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:30.423565 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391fb887-843d-4b32-aa86-32695bb3c1a3 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:30.443433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 486/950] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 16 00:47:30 2022] PUT /identity/v3/projects/7deb2c3dae214df7895b40e7339bd6ea/users/87785ccd37a642e08a0ad34a4fb1c754/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:47:32.997334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f70e03-420e-4dd1-8414-498aaffe2db8 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:32.998104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f70e03-420e-4dd1-8414-498aaffe2db8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:32.998104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f70e03-420e-4dd1-8414-498aaffe2db8 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:33.000655 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 465/951] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:47:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:47:33.007760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be45a632-8fb7-4be9-930e-9f1dec367fa1 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:33.008404 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be45a632-8fb7-4be9-930e-9f1dec367fa1 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:33.008535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be45a632-8fb7-4be9-930e-9f1dec367fa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:33.031101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-be45a632-8fb7-4be9-930e-9f1dec367fa1 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:47:33.034152 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-be45a632-8fb7-4be9-930e-9f1dec367fa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:47:33.132722 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 487/952] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:47:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:47:33.431755 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f70e03-420e-4dd1-8414-498aaffe2db8 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:33.492463 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:33.576304 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f70e03-420e-4dd1-8414-498aaffe2db8 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:33.624010 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ad61b5-dab6-42f7-b640-019a375c4dcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:33.625484 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ad61b5-dab6-42f7-b640-019a375c4dcb None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:33.625913 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ad61b5-dab6-42f7-b640-019a375c4dcb None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:33.626298 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ad61b5-dab6-42f7-b640-019a375c4dcb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:33.640498 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ad61b5-dab6-42f7-b640-019a375c4dcb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 16 00:47:33.647489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-88ad61b5-dab6-42f7-b640-019a375c4dcb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:47:33.647489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:33.647489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:33.647489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:33.647489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:33.647489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:33.647489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:33.647489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:33.647489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:33.647489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:33.647489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:33.647489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:33.647489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 16 00:47:33.647489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 16 00:47:33.647489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 16 00:47:33.647489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 16 00:47:33.649094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 16 00:47:33.649094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 16 00:47:33.649094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 16 00:47:33.649094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 16 00:47:33.649094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:33.649094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:33.649094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:33.649094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:33.649094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:33.649094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:33.649094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:33.649094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:33.649094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:33.649094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:33.649094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:33.649094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:33.650271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:33.650271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:33.650271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:33.650271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:33.650271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 16 00:47:33.650271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 16 00:47:33.650271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 16 00:47:33.650271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 16 00:47:33.650271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 16 00:47:33.650271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 16 00:47:33.650271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 16 00:47:33.650271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:33.657109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 466/953] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 16 00:47:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:33.658457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be45a632-8fb7-4be9-930e-9f1dec367fa1 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:33.663211 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:33.814106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be45a632-8fb7-4be9-930e-9f1dec367fa1 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:33.867294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffcaffa-c43b-4fe3-a19c-6b0b1e7183bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:33.867990 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffcaffa-c43b-4fe3-a19c-6b0b1e7183bd None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:33.868343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffcaffa-c43b-4fe3-a19c-6b0b1e7183bd None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:33.868581 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffcaffa-c43b-4fe3-a19c-6b0b1e7183bd None admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:33.869261 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bffcaffa-c43b-4fe3-a19c-6b0b1e7183bd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:33.869620 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bffcaffa-c43b-4fe3-a19c-6b0b1e7183bd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:33.871096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bffcaffa-c43b-4fe3-a19c-6b0b1e7183bd None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:33.885694 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 488/954] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 16 00:47:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:33.895389 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ad61b5-dab6-42f7-b640-019a375c4dcb None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:34.035347 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:34.040067 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ad61b5-dab6-42f7-b640-019a375c4dcb None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:34.087749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73dabf7e-8849-4b86-960f-8575a2864219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:34.088534 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dabf7e-8849-4b86-960f-8575a2864219 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:34.088843 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dabf7e-8849-4b86-960f-8575a2864219 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:34.089151 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dabf7e-8849-4b86-960f-8575a2864219 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:34.103080 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73dabf7e-8849-4b86-960f-8575a2864219 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:34.105053 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73dabf7e-8849-4b86-960f-8575a2864219 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:34.130429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-73dabf7e-8849-4b86-960f-8575a2864219 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:34.130429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:34.130429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:34.130429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:34.130429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:34.130429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:34.130429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:34.130429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:34.130429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:34.130429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:34.130429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:34.130429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:34.130429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:47:34.130429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:47:34.130429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:47:34.130429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:47:34.132186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:34.132186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:34.132186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:34.132186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:34.132186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:34.132186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:34.132186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:34.132186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:34.132186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:34.132186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:34.132186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:34.132186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:34.132186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:34.132186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:34.132186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:34.132186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:34.132186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:47:34.133365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:47:34.133365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:34.133365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:34.133365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 467/955] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:47:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:34.138804 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffcaffa-c43b-4fe3-a19c-6b0b1e7183bd None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:34.225232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:34.267149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffcaffa-c43b-4fe3-a19c-6b0b1e7183bd None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:34.318127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38ddc62-7fb9-4753-9696-4af5ba52b20a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:34.319270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38ddc62-7fb9-4753-9696-4af5ba52b20a None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:34.319586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38ddc62-7fb9-4753-9696-4af5ba52b20a None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:34.319975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38ddc62-7fb9-4753-9696-4af5ba52b20a None admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:34.320775 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38ddc62-7fb9-4753-9696-4af5ba52b20a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:34.321016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38ddc62-7fb9-4753-9696-4af5ba52b20a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:34.322465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38ddc62-7fb9-4753-9696-4af5ba52b20a None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:34.343018 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 489/956] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:47:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:47:34.345954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73dabf7e-8849-4b86-960f-8575a2864219 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:34.477978 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73dabf7e-8849-4b86-960f-8575a2864219 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:34.524555 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2724d8-0eed-4e7b-951d-6e18164a6ada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:34.525344 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2724d8-0eed-4e7b-951d-6e18164a6ada None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:34.525722 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2724d8-0eed-4e7b-951d-6e18164a6ada None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:34.525966 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2724d8-0eed-4e7b-951d-6e18164a6ada None admin] PATH_INFO: `/v3/users/placement` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:34.537962 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2724d8-0eed-4e7b-951d-6e18164a6ada None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:34.539898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2724d8-0eed-4e7b-951d-6e18164a6ada None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:34.556424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-2b2724d8-0eed-4e7b-951d-6e18164a6ada None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 16 00:47:34.556424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:34.556424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:34.556424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:34.556424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:34.556424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:34.556424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:34.556424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:34.556424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:34.556424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:34.556424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:34.556424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:34.556424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 16 00:47:34.556424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 16 00:47:34.556424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 16 00:47:34.556424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:34.558406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:34.560373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:34.560373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:34.560373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 16 00:47:34.560373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 16 00:47:34.560373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 16 00:47:34.560373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 16 00:47:34.560373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 16 00:47:34.560373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 16 00:47:34.560373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 16 00:47:34.560373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:34.560373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 468/957] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:47:34 2022] GET /identity/v3/users/placement => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:34.566687 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38ddc62-7fb9-4753-9696-4af5ba52b20a None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:34.705703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38ddc62-7fb9-4753-9696-4af5ba52b20a None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:34.745801 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:34.756705 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e930d79-3758-40e1-af50-18a5ded26d0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:34.757436 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e930d79-3758-40e1-af50-18a5ded26d0a None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:34.757764 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e930d79-3758-40e1-af50-18a5ded26d0a None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:34.758149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e930d79-3758-40e1-af50-18a5ded26d0a None admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:34.758983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e930d79-3758-40e1-af50-18a5ded26d0a None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:34.759339 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e930d79-3758-40e1-af50-18a5ded26d0a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:34.765046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e930d79-3758-40e1-af50-18a5ded26d0a None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:34.788194 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 490/958] 10.222.0.25 () {56 vars in 1143 bytes} [Sun Oct 16 00:47:34 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:34.794045 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2724d8-0eed-4e7b-951d-6e18164a6ada None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:34.817259 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:34.958833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2724d8-0eed-4e7b-951d-6e18164a6ada None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:35.008130 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9122d34-dde3-4fd8-9410-a267a988356d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:35.008852 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9122d34-dde3-4fd8-9410-a267a988356d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:35.009138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9122d34-dde3-4fd8-9410-a267a988356d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:35.009414 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9122d34-dde3-4fd8-9410-a267a988356d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:35.120024 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9122d34-dde3-4fd8-9410-a267a988356d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:35.122159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9122d34-dde3-4fd8-9410-a267a988356d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:35.139762 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:35.261855 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 469/959] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 16 00:47:34 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 468 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 16 00:47:35.271046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e930d79-3758-40e1-af50-18a5ded26d0a None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:35.417667 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e930d79-3758-40e1-af50-18a5ded26d0a None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:35.465658 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50febca-154c-4860-b713-53a61369951f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:35.466423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50febca-154c-4860-b713-53a61369951f None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:35.466801 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50febca-154c-4860-b713-53a61369951f None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:35.467164 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50febca-154c-4860-b713-53a61369951f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:35.479628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50febca-154c-4860-b713-53a61369951f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:35.481732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50febca-154c-4860-b713-53a61369951f None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:35.504885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-b50febca-154c-4860-b713-53a61369951f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:35.504885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:35.504885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:35.504885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:35.504885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:35.504885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:35.504885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:35.504885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:35.504885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:35.504885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:35.504885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:35.504885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:35.504885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 16 00:47:35.504885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 16 00:47:35.504885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 16 00:47:35.504885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 16 00:47:35.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:35.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:35.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:35.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:35.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:35.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:35.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:35.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:35.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:35.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:35.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:35.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:35.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:35.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:35.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:35.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:35.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 16 00:47:35.509514 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 16 00:47:35.509514 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 16 00:47:35.509514 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:47:35.509514 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 491/960] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:47:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:35.512745 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9122d34-dde3-4fd8-9410-a267a988356d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:35.674017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9122d34-dde3-4fd8-9410-a267a988356d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:35.700071 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:35.739517 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f4048a-16eb-4252-bf88-986810f213a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:35.740584 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f4048a-16eb-4252-bf88-986810f213a8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:35.741060 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f4048a-16eb-4252-bf88-986810f213a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:35.741550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f4048a-16eb-4252-bf88-986810f213a8 None admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:35.742649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f4048a-16eb-4252-bf88-986810f213a8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:35.743195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f4048a-16eb-4252-bf88-986810f213a8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:35.745729 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f4048a-16eb-4252-bf88-986810f213a8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:35.763621 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 470/961] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 16 00:47:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:47:35.771978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50febca-154c-4860-b713-53a61369951f None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:35.832183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:35.932594 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50febca-154c-4860-b713-53a61369951f None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:35.974363 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b16cf0-8c14-474f-b654-5a8a1e7f76af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:35.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b16cf0-8c14-474f-b654-5a8a1e7f76af None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:35.975377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b16cf0-8c14-474f-b654-5a8a1e7f76af None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:35.975653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b16cf0-8c14-474f-b654-5a8a1e7f76af None admin] PATH_INFO: `/v3/projects/service` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:35.983959 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b16cf0-8c14-474f-b654-5a8a1e7f76af None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:35.983959 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b16cf0-8c14-474f-b654-5a8a1e7f76af None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:35.998456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application [None req-d5b16cf0-8c14-474f-b654-5a8a1e7f76af None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:47:35.998456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:35.998456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:35.998456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:35.998456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:35.998456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:35.998456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:35.998456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:35.998456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:35.998456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:35.998456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:35.998456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:35.998456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 16 00:47:35.998456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 16 00:47:35.998456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 16 00:47:36.005129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 16 00:47:36.005129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:36.005129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:36.005129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:36.005129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:36.005129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:36.005129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:36.005129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:36.005129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:36.005129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:36.005129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:36.005129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:36.005129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:36.005129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:36.005129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:36.005129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:36.005129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:36.007329 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 16 00:47:36.007329 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 16 00:47:36.007329 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 16 00:47:36.007329 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 16 00:47:36.007329 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 16 00:47:36.007329 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 16 00:47:36.007329 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 16 00:47:36.007329 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: ERROR keystone.server.flask.application Oct 16 00:47:36.007329 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 492/962] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 16 00:47:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:36.008608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f4048a-16eb-4252-bf88-986810f213a8 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:36.127090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f4048a-16eb-4252-bf88-986810f213a8 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:36.168275 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40bf00a9-67c2-48bc-bdea-aefe96f86aa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:36.169014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bf00a9-67c2-48bc-bdea-aefe96f86aa8 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:36.169403 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bf00a9-67c2-48bc-bdea-aefe96f86aa8 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:36.169676 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bf00a9-67c2-48bc-bdea-aefe96f86aa8 None admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:36.170803 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40bf00a9-67c2-48bc-bdea-aefe96f86aa8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:36.171150 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40bf00a9-67c2-48bc-bdea-aefe96f86aa8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:36.173144 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40bf00a9-67c2-48bc-bdea-aefe96f86aa8 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:36.194925 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 471/963] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 16 00:47:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:36.205084 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b16cf0-8c14-474f-b654-5a8a1e7f76af None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:36.330397 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:36.340976 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b16cf0-8c14-474f-b654-5a8a1e7f76af None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:36.380941 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a99b2e5-4288-467e-a003-2f3d8c487a14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:36.381489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a99b2e5-4288-467e-a003-2f3d8c487a14 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:36.381770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a99b2e5-4288-467e-a003-2f3d8c487a14 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:36.382048 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a99b2e5-4288-467e-a003-2f3d8c487a14 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:36.382797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a99b2e5-4288-467e-a003-2f3d8c487a14 None admin] RBAC: Adding query filter params (role.id=c5cd777d70a649028f970fd986dc484f, scope.project.id=7deb2c3dae214df7895b40e7339bd6ea, user.id=87785ccd37a642e08a0ad34a4fb1c754) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:36.383146 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a99b2e5-4288-467e-a003-2f3d8c487a14 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:36.384615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a99b2e5-4288-467e-a003-2f3d8c487a14 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:36.392035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 493/964] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 16 00:47:36 2022] GET /identity/v3/role_assignments?user.id=87785ccd37a642e08a0ad34a4fb1c754&scope.project.id=7deb2c3dae214df7895b40e7339bd6ea&role.id=c5cd777d70a649028f970fd986dc484f => generated 608 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:47:38.662816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4221841c-6a72-4f2b-aa49-4e297cce8b0b None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:38.663439 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4221841c-6a72-4f2b-aa49-4e297cce8b0b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:38.663589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4221841c-6a72-4f2b-aa49-4e297cce8b0b None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:38.665212 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 472/965] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:47:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:47:38.672826 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd666430-41fa-4b77-9c4b-c790b8d52504 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:38.673276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd666430-41fa-4b77-9c4b-c790b8d52504 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:38.673387 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd666430-41fa-4b77-9c4b-c790b8d52504 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:38.692192 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-bd666430-41fa-4b77-9c4b-c790b8d52504 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:47:38.693311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-bd666430-41fa-4b77-9c4b-c790b8d52504 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:47:38.760141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 494/966] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:47:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:47:39.030121 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4221841c-6a72-4f2b-aa49-4e297cce8b0b None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:39.060492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:39.129401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4221841c-6a72-4f2b-aa49-4e297cce8b0b None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:39.161656 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4fb394-f47c-4190-a0d1-6d92706f110d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:39.162274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4fb394-f47c-4190-a0d1-6d92706f110d None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:39.162547 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4fb394-f47c-4190-a0d1-6d92706f110d None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:39.162814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4fb394-f47c-4190-a0d1-6d92706f110d None admin] PATH_INFO: `/v3/services/placement` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:39.163415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4fb394-f47c-4190-a0d1-6d92706f110d None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:39.164783 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4fb394-f47c-4190-a0d1-6d92706f110d None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:39.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-da4fb394-f47c-4190-a0d1-6d92706f110d None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 16 00:47:39.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:39.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:47:39.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:47:39.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:47:39.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:47:39.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:47:39.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:47:39.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 16 00:47:39.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:39.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:47:39.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:39.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:39.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:39.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:39.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:39.173817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:39.175159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:39.175159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:39.175159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:39.175159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:39.175159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:39.175159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:39.175159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:47:39.175159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:47:39.175159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:47:39.175159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:47:39.175159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:39.175159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:39.175159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:39.175159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:39.175159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:39.175159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:39.176242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:39.176242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:39.176242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:39.176242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:39.176242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:39.176242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:39.176242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:39.176242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:39.176242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:39.176242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:39.176242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:47:39.176242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:47:39.176242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 16 00:47:39.176242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:39.176242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 473/967] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 16 00:47:39 2022] GET /identity/v3/services/placement => generated 90 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:39.181064 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd666430-41fa-4b77-9c4b-c790b8d52504 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:39.275710 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:39.285888 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd666430-41fa-4b77-9c4b-c790b8d52504 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:39.320263 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea1b912-9193-43ca-8799-879b7215d1de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:39.320875 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea1b912-9193-43ca-8799-879b7215d1de None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:39.321157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea1b912-9193-43ca-8799-879b7215d1de None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:39.321432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea1b912-9193-43ca-8799-879b7215d1de None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:39.322059 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea1b912-9193-43ca-8799-879b7215d1de None admin] RBAC: Adding query filter params (name=placement) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:39.322363 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea1b912-9193-43ca-8799-879b7215d1de None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:39.323759 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea1b912-9193-43ca-8799-879b7215d1de None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:39.329057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 495/968] 10.222.0.25 () {56 vars in 1116 bytes} [Sun Oct 16 00:47:39 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:39.334810 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4fb394-f47c-4190-a0d1-6d92706f110d None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:39.429332 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:39.435600 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4fb394-f47c-4190-a0d1-6d92706f110d None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:39.469567 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1c5fb9-8012-48a4-a09f-0cd011909ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:39.470172 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1c5fb9-8012-48a4-a09f-0cd011909ab4 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:39.470456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1c5fb9-8012-48a4-a09f-0cd011909ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:39.470734 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1c5fb9-8012-48a4-a09f-0cd011909ab4 None admin] PATH_INFO: `/v3/services` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:39.471389 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc1c5fb9-8012-48a4-a09f-0cd011909ab4 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:39.471698 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc1c5fb9-8012-48a4-a09f-0cd011909ab4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:39.473078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc1c5fb9-8012-48a4-a09f-0cd011909ab4 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:39.477399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 474/969] 10.222.0.25 () {56 vars in 1116 bytes} [Sun Oct 16 00:47:39 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:41.665225 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d77b6d-cf8f-4f0d-9479-59a2b9f0cec5 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:41.665781 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d77b6d-cf8f-4f0d-9479-59a2b9f0cec5 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:41.665781 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d77b6d-cf8f-4f0d-9479-59a2b9f0cec5 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:41.667231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 496/970] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:47:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:47:41.673122 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e107757-f0d0-44bc-a51e-f3a44b982fd4 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:41.673540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e107757-f0d0-44bc-a51e-f3a44b982fd4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:41.673654 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e107757-f0d0-44bc-a51e-f3a44b982fd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:41.691703 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3e107757-f0d0-44bc-a51e-f3a44b982fd4 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:47:41.692828 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3e107757-f0d0-44bc-a51e-f3a44b982fd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:47:41.759608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 475/971] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:47:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:47:42.031723 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d77b6d-cf8f-4f0d-9479-59a2b9f0cec5 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:42.132245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d77b6d-cf8f-4f0d-9479-59a2b9f0cec5 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:42.164870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1931b3ce-343e-423c-b023-35f82de14fcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:42.165474 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931b3ce-343e-423c-b023-35f82de14fcb None admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:42.165755 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931b3ce-343e-423c-b023-35f82de14fcb None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:42.166027 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1931b3ce-343e-423c-b023-35f82de14fcb None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:42.166666 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1931b3ce-343e-423c-b023-35f82de14fcb None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:42.168060 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1931b3ce-343e-423c-b023-35f82de14fcb None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:42.184033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 497/972] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 16 00:47:42 2022] POST /identity/v3/services => generated 246 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:47:44.378345 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a8da15-14a7-4fa5-8689-2e8cfb86b35c None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:44.379034 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a8da15-14a7-4fa5-8689-2e8cfb86b35c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:44.379034 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a8da15-14a7-4fa5-8689-2e8cfb86b35c None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:44.380256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 476/973] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:47:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:47:44.385520 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba453c8-ca7b-4302-8335-f7493940beca None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:44.385892 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba453c8-ca7b-4302-8335-f7493940beca None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:44.386080 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba453c8-ca7b-4302-8335-f7493940beca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:44.405000 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-6ba453c8-ca7b-4302-8335-f7493940beca None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:47:44.406807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6ba453c8-ca7b-4302-8335-f7493940beca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:47:44.416554 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:44.482384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 498/974] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:47:44 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:47:44.751715 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a8da15-14a7-4fa5-8689-2e8cfb86b35c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:44.845098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:44.857969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a8da15-14a7-4fa5-8689-2e8cfb86b35c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:44.899348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8756b84-cf89-4bfc-ac64-356e1fb30e34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:44.899979 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8756b84-cf89-4bfc-ac64-356e1fb30e34 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:44.900260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8756b84-cf89-4bfc-ac64-356e1fb30e34 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:44.900532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8756b84-cf89-4bfc-ac64-356e1fb30e34 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:44.901112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8756b84-cf89-4bfc-ac64-356e1fb30e34 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:44.902485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8756b84-cf89-4bfc-ac64-356e1fb30e34 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:44.911260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-c8756b84-cf89-4bfc-ac64-356e1fb30e34 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 16 00:47:44.911260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:44.911260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:47:44.911260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:47:44.911260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:47:44.911260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:47:44.911260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:47:44.911260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:47:44.911260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 16 00:47:44.911260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:44.911260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:47:44.911260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:44.911260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:44.911260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:44.911260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:44.911260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:44.911260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:44.912447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:44.912447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:44.912447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:44.912447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:44.912447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:44.912447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:44.912447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:47:44.912447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:47:44.912447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:47:44.912447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:47:44.912447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:44.912447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:44.912447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:44.912447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:44.912447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:44.912447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:44.913589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:44.913589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:44.913589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:44.913589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:44.913589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:44.913589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:44.913589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:44.913589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:44.913589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:44.913589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:44.913589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:47:44.913589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:47:44.913589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 16 00:47:44.913589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:44.913589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 477/975] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 16 00:47:44 2022] GET /identity/v3/services/placement => generated 90 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:44.918420 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba453c8-ca7b-4302-8335-f7493940beca None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:45.016412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba453c8-ca7b-4302-8335-f7493940beca None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:45.049998 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4d5e25-5c2b-4270-8862-e78810476c6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:45.050609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4d5e25-5c2b-4270-8862-e78810476c6c None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:45.050919 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4d5e25-5c2b-4270-8862-e78810476c6c None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:45.051202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4d5e25-5c2b-4270-8862-e78810476c6c None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:45.051833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4d5e25-5c2b-4270-8862-e78810476c6c None admin] RBAC: Adding query filter params (name=placement) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:45.052142 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4d5e25-5c2b-4270-8862-e78810476c6c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:45.053521 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4d5e25-5c2b-4270-8862-e78810476c6c None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:45.058653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 499/976] 10.222.0.25 () {56 vars in 1116 bytes} [Sun Oct 16 00:47:44 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:45.065226 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8756b84-cf89-4bfc-ac64-356e1fb30e34 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:45.165531 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8756b84-cf89-4bfc-ac64-356e1fb30e34 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:45.198842 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95650cee-f473-48c4-96ad-087a7bc20a04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:45.199467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95650cee-f473-48c4-96ad-087a7bc20a04 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:45.199745 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95650cee-f473-48c4-96ad-087a7bc20a04 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:45.200014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95650cee-f473-48c4-96ad-087a7bc20a04 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:45.200706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95650cee-f473-48c4-96ad-087a7bc20a04 None admin] RBAC: Adding query filter params (interface=public, service_id=9f09f670bf4149c4b75ff4b1788a2ed8, region_id=RegionOne) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:45.201010 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95650cee-f473-48c4-96ad-087a7bc20a04 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:45.202415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95650cee-f473-48c4-96ad-087a7bc20a04 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:45.207630 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 478/977] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Oct 16 00:47:45 2022] GET /identity/v3/endpoints?service_id=9f09f670bf4149c4b75ff4b1788a2ed8&interface=public®ion_id=RegionOne => generated 194 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:45.213291 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4d5e25-5c2b-4270-8862-e78810476c6c None admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:45.217131 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:45.312515 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4d5e25-5c2b-4270-8862-e78810476c6c None admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:45.346451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c10aac3-5fa8-4993-af86-b09c4a812158 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:45.347143 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c10aac3-5fa8-4993-af86-b09c4a812158 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:45.347437 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c10aac3-5fa8-4993-af86-b09c4a812158 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:45.347714 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c10aac3-5fa8-4993-af86-b09c4a812158 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:45.348325 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c10aac3-5fa8-4993-af86-b09c4a812158 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:45.349695 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c10aac3-5fa8-4993-af86-b09c4a812158 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:45.356473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 500/978] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 16 00:47:45 2022] GET /identity/v3/services => generated 2191 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:47:47.524290 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3645215b-df5b-4c95-bdfd-1f21a52e5659 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:47.524812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3645215b-df5b-4c95-bdfd-1f21a52e5659 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:47.524923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3645215b-df5b-4c95-bdfd-1f21a52e5659 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:47.526175 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 479/979] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:47:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:47:47.531693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f961bf60-c897-4d15-a371-682ba3e25fcf None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:47.532142 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f961bf60-c897-4d15-a371-682ba3e25fcf None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:47.532235 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f961bf60-c897-4d15-a371-682ba3e25fcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:47.550572 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-f961bf60-c897-4d15-a371-682ba3e25fcf None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:47:47.551782 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-f961bf60-c897-4d15-a371-682ba3e25fcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:47:47.613549 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:47.619517 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 501/980] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:47:47 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 16 00:47:47.899359 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3645215b-df5b-4c95-bdfd-1f21a52e5659 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:47.918657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:47.999351 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3645215b-df5b-4c95-bdfd-1f21a52e5659 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:48.032647 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb83147-124c-4da0-8683-de20ea2b1597 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:48.033271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb83147-124c-4da0-8683-de20ea2b1597 None admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:48.033550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb83147-124c-4da0-8683-de20ea2b1597 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:48.033822 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb83147-124c-4da0-8683-de20ea2b1597 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:48.034397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb83147-124c-4da0-8683-de20ea2b1597 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:48.035791 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb83147-124c-4da0-8683-de20ea2b1597 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:48.044462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application [None req-cdb83147-124c-4da0-8683-de20ea2b1597 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 16 00:47:48.044462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:48.044462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 16 00:47:48.044462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 16 00:47:48.044462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 16 00:47:48.044462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 16 00:47:48.044462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 16 00:47:48.044462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:47:48.044462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 16 00:47:48.044462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:48.044462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 16 00:47:48.044462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:48.044462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 16 00:47:48.044462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 16 00:47:48.044462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 16 00:47:48.044462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 16 00:47:48.044462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 16 00:47:48.045618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 16 00:47:48.045618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 16 00:47:48.045618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 16 00:47:48.045618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 16 00:47:48.045618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 16 00:47:48.045618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 16 00:47:48.045618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 16 00:47:48.045618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 16 00:47:48.045618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 16 00:47:48.045618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 16 00:47:48.045618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 16 00:47:48.045618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 16 00:47:48.045618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 16 00:47:48.045618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 16 00:47:48.045618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 16 00:47:48.045618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.get_or_create( Oct 16 00:47:48.046695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 16 00:47:48.046695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application with Lock( Oct 16 00:47:48.046695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 16 00:47:48.046695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self._enter() Oct 16 00:47:48.046695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 16 00:47:48.046695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 16 00:47:48.046695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 16 00:47:48.046695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application return self.creator() Oct 16 00:47:48.046695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 16 00:47:48.046695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application created_value = creator( Oct 16 00:47:48.046695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 16 00:47:48.046695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 16 00:47:48.046695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 16 00:47:48.046695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: ERROR keystone.server.flask.application Oct 16 00:47:48.046695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 480/981] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 16 00:47:47 2022] GET /identity/v3/services/placement => generated 90 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 16 00:47:48.051682 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f961bf60-c897-4d15-a371-682ba3e25fcf None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:48.158487 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f961bf60-c897-4d15-a371-682ba3e25fcf None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:48.192048 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f03229-501a-4b84-9f8f-599acd35ef97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:48.192668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f03229-501a-4b84-9f8f-599acd35ef97 None admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:48.192952 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f03229-501a-4b84-9f8f-599acd35ef97 None admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:48.193230 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f03229-501a-4b84-9f8f-599acd35ef97 None admin] PATH_INFO: `/v3/services` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:48.193862 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f03229-501a-4b84-9f8f-599acd35ef97 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:47:48.194170 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f03229-501a-4b84-9f8f-599acd35ef97 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:48.195576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f03229-501a-4b84-9f8f-599acd35ef97 None admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:48.200733 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 502/982] 10.222.0.25 () {56 vars in 1116 bytes} [Sun Oct 16 00:47:48 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:47:48.207607 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb83147-124c-4da0-8683-de20ea2b1597 None admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:47:48.292942 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:48.311064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb83147-124c-4da0-8683-de20ea2b1597 None admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:47:48.346041 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803f2a34-251c-44d2-b10b-831ef140a397 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:47:48.346646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803f2a34-251c-44d2-b10b-831ef140a397 None admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:47:48.346953 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803f2a34-251c-44d2-b10b-831ef140a397 None admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:47:48.347227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803f2a34-251c-44d2-b10b-831ef140a397 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:47:48.347880 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803f2a34-251c-44d2-b10b-831ef140a397 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:47:48.349247 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803f2a34-251c-44d2-b10b-831ef140a397 None admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:47:48.368368 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 481/983] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:47:48 2022] POST /identity/v3/endpoints => generated 331 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:47:58.197872 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=69 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:47:58.364548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:48:57.592292 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1525642-3c03-495b-b365-dfbe0d63ba53 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:48:57.593454 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1525642-3c03-495b-b365-dfbe0d63ba53 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:48:57.594087 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1525642-3c03-495b-b365-dfbe0d63ba53 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:48:57.597076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 503/984] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:48:57 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:48:57.602566 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e73859-020a-49d3-8bdc-a05fbc9390c2 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:48:57.603041 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e73859-020a-49d3-8bdc-a05fbc9390c2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:48:57.603041 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e73859-020a-49d3-8bdc-a05fbc9390c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:48:57.749047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-91e73859-020a-49d3-8bdc-a05fbc9390c2 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:48:57.751625 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-91e73859-020a-49d3-8bdc-a05fbc9390c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:48:57.871991 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 482/985] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:48:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:48:58.198464 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c343743-c9fa-4ded-9071-71f547300af5 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:48:58.199560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c343743-c9fa-4ded-9071-71f547300af5 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:48:58.201474 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c343743-c9fa-4ded-9071-71f547300af5 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:48:58.206061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 504/986] 10.222.0.25 () {54 vars in 964 bytes} [Sun Oct 16 00:48:58 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:48:58.214499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9e27c5-a8ce-49d7-a383-902085a5006e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:48:58.215980 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9e27c5-a8ce-49d7-a383-902085a5006e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:48:58.216829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9e27c5-a8ce-49d7-a383-902085a5006e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:48:58.335814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-1a9e27c5-a8ce-49d7-a383-902085a5006e None None] MFA Rules not processed for user `f4eb1a640186411097cb1e3bb9e92020`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:48:58.337133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1a9e27c5-a8ce-49d7-a383-902085a5006e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:48:58.367554 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:48:58.456291 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 483/987] 10.222.0.25 () {58 vars in 1061 bytes} [Sun Oct 16 00:48:58 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:48:58.465037 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c343743-c9fa-4ded-9071-71f547300af5 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:48:58.466828 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-0c343743-c9fa-4ded-9071-71f547300af5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:48:58.607223 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c343743-c9fa-4ded-9071-71f547300af5 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:48:58.703455 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4f02d8-e239-40d0-9804-2394579e4ca6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:48:58.704047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4f02d8-e239-40d0-9804-2394579e4ca6 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:48:58.704335 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4f02d8-e239-40d0-9804-2394579e4ca6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:48:58.704613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4f02d8-e239-40d0-9804-2394579e4ca6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:48:58.706304 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-2f4f02d8-e239-40d0-9804-2394579e4ca6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:48:58.770697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:48:58.837922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4f02d8-e239-40d0-9804-2394579e4ca6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:48:58.839781 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4f02d8-e239-40d0-9804-2394579e4ca6 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:48:58.864148 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 505/988] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:48:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:49:01.704439 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772550ff-8d9b-4fec-afa9-9a9694fc5816 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:01.705017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772550ff-8d9b-4fec-afa9-9a9694fc5816 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:01.705017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772550ff-8d9b-4fec-afa9-9a9694fc5816 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:01.706334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 484/989] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:49:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:49:01.711714 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4f64e1-0b04-433a-ad47-56d4bbe5cfb8 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:01.712223 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4f64e1-0b04-433a-ad47-56d4bbe5cfb8 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:01.712336 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4f64e1-0b04-433a-ad47-56d4bbe5cfb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:01.806400 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-1d4f64e1-0b04-433a-ad47-56d4bbe5cfb8 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:49:01.808048 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-1d4f64e1-0b04-433a-ad47-56d4bbe5cfb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:01.920740 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 506/990] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:49:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:49:02.215346 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772550ff-8d9b-4fec-afa9-9a9694fc5816 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:49:02.216957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-772550ff-8d9b-4fec-afa9-9a9694fc5816 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:02.335309 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772550ff-8d9b-4fec-afa9-9a9694fc5816 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:49:02.436373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:02.438918 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da6cc66-726a-4c64-8324-2f84a265b7fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:49:02.439603 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da6cc66-726a-4c64-8324-2f84a265b7fe service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:02.440089 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da6cc66-726a-4c64-8324-2f84a265b7fe service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:02.440426 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da6cc66-726a-4c64-8324-2f84a265b7fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:02.442538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4da6cc66-726a-4c64-8324-2f84a265b7fe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:02.560266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da6cc66-726a-4c64-8324-2f84a265b7fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:49:02.561724 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da6cc66-726a-4c64-8324-2f84a265b7fe service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:49:02.581807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 485/991] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:49:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:49:05.147558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a71adb7-fe89-46bc-9bed-ac0b43e406e5 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:05.148131 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a71adb7-fe89-46bc-9bed-ac0b43e406e5 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:05.148131 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a71adb7-fe89-46bc-9bed-ac0b43e406e5 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:05.149454 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 507/992] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:49:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:49:05.156153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a722f691-fbd8-4c3d-adfe-2197c0491fa5 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:05.156676 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a722f691-fbd8-4c3d-adfe-2197c0491fa5 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:05.156891 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a722f691-fbd8-4c3d-adfe-2197c0491fa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:05.240603 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a722f691-fbd8-4c3d-adfe-2197c0491fa5 None None] MFA Rules not processed for user `620e6f7dab50405d9bebd872ceb59dc7`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:49:05.241837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a722f691-fbd8-4c3d-adfe-2197c0491fa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:05.294731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:05.340242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 486/993] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:49:05 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:49:05.647231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b53f8d8-007c-4887-9449-77423e6882bb None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:05.647500 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b53f8d8-007c-4887-9449-77423e6882bb None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:05.647778 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b53f8d8-007c-4887-9449-77423e6882bb None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:05.649132 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 508/994] 10.222.0.25 () {54 vars in 964 bytes} [Sun Oct 16 00:49:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:49:05.655305 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16dca32-6d5c-48ce-89c9-47c81ac7d0d6 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:05.655517 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16dca32-6d5c-48ce-89c9-47c81ac7d0d6 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:05.655717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16dca32-6d5c-48ce-89c9-47c81ac7d0d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:05.735375 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c16dca32-6d5c-48ce-89c9-47c81ac7d0d6 None None] MFA Rules not processed for user `f4eb1a640186411097cb1e3bb9e92020`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:49:05.736495 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c16dca32-6d5c-48ce-89c9-47c81ac7d0d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:05.812893 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 487/995] 10.222.0.25 () {58 vars in 1061 bytes} [Sun Oct 16 00:49:05 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:49:05.821995 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b53f8d8-007c-4887-9449-77423e6882bb None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:49:05.824177 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8b53f8d8-007c-4887-9449-77423e6882bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:05.836048 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:05.933288 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b53f8d8-007c-4887-9449-77423e6882bb None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:49:06.009936 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b1b518-57c5-495e-b24e-4c1d0dc351d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:49:06.010746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b1b518-57c5-495e-b24e-4c1d0dc351d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:06.011137 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b1b518-57c5-495e-b24e-4c1d0dc351d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:06.011491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b1b518-57c5-495e-b24e-4c1d0dc351d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:06.013086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-57b1b518-57c5-495e-b24e-4c1d0dc351d4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:06.079200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:06.129037 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b1b518-57c5-495e-b24e-4c1d0dc351d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:49:06.130501 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b1b518-57c5-495e-b24e-4c1d0dc351d4 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:49:06.151632 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 509/996] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:49:05 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:49:09.262204 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d7145b-49a6-4831-8ee1-5166321b7c0c None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:09.262788 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d7145b-49a6-4831-8ee1-5166321b7c0c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:09.262788 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d7145b-49a6-4831-8ee1-5166321b7c0c None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:09.264191 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 488/997] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:49:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:49:09.270182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ad82c7-cd2f-4275-8e39-51d677667725 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:09.270661 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ad82c7-cd2f-4275-8e39-51d677667725 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:09.270769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ad82c7-cd2f-4275-8e39-51d677667725 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:09.355196 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-26ad82c7-cd2f-4275-8e39-51d677667725 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:49:09.357331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-26ad82c7-cd2f-4275-8e39-51d677667725 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:09.483215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 510/998] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:49:09 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:49:09.774226 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d7145b-49a6-4831-8ee1-5166321b7c0c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:49:09.775973 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-34d7145b-49a6-4831-8ee1-5166321b7c0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:09.835505 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:09.903239 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d7145b-49a6-4831-8ee1-5166321b7c0c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:49:09.981243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fef7c87-7d63-4c0b-8f4e-9f16376bcbbd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:49:09.981860 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fef7c87-7d63-4c0b-8f4e-9f16376bcbbd service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:09.982140 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fef7c87-7d63-4c0b-8f4e-9f16376bcbbd service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:09.982413 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fef7c87-7d63-4c0b-8f4e-9f16376bcbbd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:09.984094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4fef7c87-7d63-4c0b-8f4e-9f16376bcbbd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:10.080941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:10.096176 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fef7c87-7d63-4c0b-8f4e-9f16376bcbbd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:49:10.097638 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fef7c87-7d63-4c0b-8f4e-9f16376bcbbd service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:49:10.116702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 489/999] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:49:09 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:49:13.014674 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c6303f-590d-4cc3-a89e-49d7a480474d None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:13.015571 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c6303f-590d-4cc3-a89e-49d7a480474d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:13.015571 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c6303f-590d-4cc3-a89e-49d7a480474d None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:13.016797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 511/1000] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:49:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:49:13.022343 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35cd0cf-299b-4fcf-a6d6-876710798a95 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:13.022808 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35cd0cf-299b-4fcf-a6d6-876710798a95 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:13.022953 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35cd0cf-299b-4fcf-a6d6-876710798a95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:13.097251 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c35cd0cf-299b-4fcf-a6d6-876710798a95 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:49:13.098606 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c35cd0cf-299b-4fcf-a6d6-876710798a95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:13.220629 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 490/1001] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:49:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:49:13.510307 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c6303f-590d-4cc3-a89e-49d7a480474d None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:49:13.512244 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-89c6303f-590d-4cc3-a89e-49d7a480474d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:13.534974 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:13.643304 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c6303f-590d-4cc3-a89e-49d7a480474d None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:49:13.726260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc43b55-026d-4d75-94a2-fda22eb57784 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:49:13.726850 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc43b55-026d-4d75-94a2-fda22eb57784 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:13.727166 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc43b55-026d-4d75-94a2-fda22eb57784 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:13.727442 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc43b55-026d-4d75-94a2-fda22eb57784 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:13.729700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8fc43b55-026d-4d75-94a2-fda22eb57784 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:13.808832 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:13.851200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc43b55-026d-4d75-94a2-fda22eb57784 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:49:13.852651 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc43b55-026d-4d75-94a2-fda22eb57784 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:49:13.873529 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 512/1002] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:49:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:49:16.177581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa350fe-90d0-4670-96c4-e49c4dc8164a None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:16.178157 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa350fe-90d0-4670-96c4-e49c4dc8164a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:16.178157 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa350fe-90d0-4670-96c4-e49c4dc8164a None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:16.179523 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 491/1003] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:49:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:49:16.185761 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1a7144-ae00-426b-907e-6a76298d5f3d None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:16.186275 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1a7144-ae00-426b-907e-6a76298d5f3d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:16.186384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1a7144-ae00-426b-907e-6a76298d5f3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:16.263770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-2f1a7144-ae00-426b-907e-6a76298d5f3d None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:49:16.265712 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-2f1a7144-ae00-426b-907e-6a76298d5f3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:16.381765 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 513/1004] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:49:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:49:16.666160 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa350fe-90d0-4670-96c4-e49c4dc8164a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:49:16.667925 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7fa350fe-90d0-4670-96c4-e49c4dc8164a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:16.740789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:16.794512 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa350fe-90d0-4670-96c4-e49c4dc8164a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:49:16.894538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c28e6b-26d8-474e-8ea5-678ddf7de4d3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:49:16.895194 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c28e6b-26d8-474e-8ea5-678ddf7de4d3 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:16.895649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c28e6b-26d8-474e-8ea5-678ddf7de4d3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:16.895925 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c28e6b-26d8-474e-8ea5-678ddf7de4d3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:16.897594 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-15c28e6b-26d8-474e-8ea5-678ddf7de4d3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:17.018070 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:17.020773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c28e6b-26d8-474e-8ea5-678ddf7de4d3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:49:17.022320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c28e6b-26d8-474e-8ea5-678ddf7de4d3 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:49:17.042753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 492/1005] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:49:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:49:19.319152 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da668730-8cc5-4d05-8382-ddbc9609a6bc None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:19.319709 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da668730-8cc5-4d05-8382-ddbc9609a6bc None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:19.319709 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da668730-8cc5-4d05-8382-ddbc9609a6bc None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:19.321041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 514/1006] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:49:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:49:19.327086 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb78ba-b755-4d5c-b5ac-78abdbf7f922 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:19.327693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb78ba-b755-4d5c-b5ac-78abdbf7f922 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:19.327693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb78ba-b755-4d5c-b5ac-78abdbf7f922 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:19.402228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b5cb78ba-b755-4d5c-b5ac-78abdbf7f922 None None] MFA Rules not processed for user `620e6f7dab50405d9bebd872ceb59dc7`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:49:19.403508 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b5cb78ba-b755-4d5c-b5ac-78abdbf7f922 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:19.507638 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 493/1007] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:49:19 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:49:19.806297 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da668730-8cc5-4d05-8382-ddbc9609a6bc None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:49:19.808273 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-da668730-8cc5-4d05-8382-ddbc9609a6bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:19.848681 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:19.949381 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da668730-8cc5-4d05-8382-ddbc9609a6bc None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:49:20.066261 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46df79fc-1570-447e-93e3-0743d86df060 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:49:20.066951 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46df79fc-1570-447e-93e3-0743d86df060 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:20.067193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46df79fc-1570-447e-93e3-0743d86df060 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:20.067463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46df79fc-1570-447e-93e3-0743d86df060 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:20.069091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-46df79fc-1570-447e-93e3-0743d86df060 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:20.154265 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:20.190227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46df79fc-1570-447e-93e3-0743d86df060 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:49:20.191657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46df79fc-1570-447e-93e3-0743d86df060 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:49:20.214439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 515/1008] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:49:19 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:49:22.642407 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc47eae2-f53c-411d-91a8-10629645a37e None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:22.642990 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc47eae2-f53c-411d-91a8-10629645a37e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:22.642990 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc47eae2-f53c-411d-91a8-10629645a37e None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:22.644379 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 494/1009] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:49:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:49:22.649766 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afa06ad-1a2d-4f23-8486-e41373616ae7 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:22.650889 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afa06ad-1a2d-4f23-8486-e41373616ae7 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:22.650889 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afa06ad-1a2d-4f23-8486-e41373616ae7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:22.723554 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-6afa06ad-1a2d-4f23-8486-e41373616ae7 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:49:22.724702 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6afa06ad-1a2d-4f23-8486-e41373616ae7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:22.835088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 516/1010] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:49:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:49:23.126375 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc47eae2-f53c-411d-91a8-10629645a37e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:49:23.127973 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-dc47eae2-f53c-411d-91a8-10629645a37e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:23.210108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:23.250715 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc47eae2-f53c-411d-91a8-10629645a37e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:49:23.331103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79704097-1757-4bb4-a6dc-6b0477d16f98 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:49:23.331883 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79704097-1757-4bb4-a6dc-6b0477d16f98 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:23.332168 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79704097-1757-4bb4-a6dc-6b0477d16f98 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:23.332437 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79704097-1757-4bb4-a6dc-6b0477d16f98 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:23.334051 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-79704097-1757-4bb4-a6dc-6b0477d16f98 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:23.454580 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79704097-1757-4bb4-a6dc-6b0477d16f98 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:49:23.456072 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79704097-1757-4bb4-a6dc-6b0477d16f98 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:49:23.468287 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:23.479174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 495/1011] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:49:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:49:25.722853 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560cd097-46e9-49f6-9760-46c6925457dd None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:25.723395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560cd097-46e9-49f6-9760-46c6925457dd None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:25.723395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560cd097-46e9-49f6-9760-46c6925457dd None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:25.724713 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 517/1012] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:49:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:49:25.730545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121919e7-b48e-4dbe-b265-17bdd122760c None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:25.731008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121919e7-b48e-4dbe-b265-17bdd122760c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:25.731140 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121919e7-b48e-4dbe-b265-17bdd122760c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:25.798585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-121919e7-b48e-4dbe-b265-17bdd122760c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:49:25.799872 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-121919e7-b48e-4dbe-b265-17bdd122760c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:25.900019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 496/1013] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:49:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:49:26.191515 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560cd097-46e9-49f6-9760-46c6925457dd None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:49:26.193260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-560cd097-46e9-49f6-9760-46c6925457dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:26.241363 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:26.313422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560cd097-46e9-49f6-9760-46c6925457dd None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:49:26.396949 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb58a1f-176e-47eb-b88e-ccf52e2d99e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:49:26.397563 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb58a1f-176e-47eb-b88e-ccf52e2d99e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:26.397845 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb58a1f-176e-47eb-b88e-ccf52e2d99e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:26.398122 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb58a1f-176e-47eb-b88e-ccf52e2d99e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:26.400372 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-2fb58a1f-176e-47eb-b88e-ccf52e2d99e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:26.486404 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:26.514303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb58a1f-176e-47eb-b88e-ccf52e2d99e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:49:26.515795 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb58a1f-176e-47eb-b88e-ccf52e2d99e9 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:49:26.534736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 518/1014] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:49:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:49:29.076908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4281aca7-3491-4e1c-9e5d-b7e8aebe255e None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:29.077497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4281aca7-3491-4e1c-9e5d-b7e8aebe255e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:29.077497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4281aca7-3491-4e1c-9e5d-b7e8aebe255e None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:29.078826 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 497/1015] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:49:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:49:29.084500 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b93143-a14b-4391-a0ea-f41f02c5b21f None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:29.084949 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b93143-a14b-4391-a0ea-f41f02c5b21f None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:29.085063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b93143-a14b-4391-a0ea-f41f02c5b21f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:29.155833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-96b93143-a14b-4391-a0ea-f41f02c5b21f None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:49:29.157100 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-96b93143-a14b-4391-a0ea-f41f02c5b21f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:29.262095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 519/1016] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:49:29 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:49:29.553511 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4281aca7-3491-4e1c-9e5d-b7e8aebe255e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:49:29.555115 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4281aca7-3491-4e1c-9e5d-b7e8aebe255e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:29.643276 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:29.672994 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4281aca7-3491-4e1c-9e5d-b7e8aebe255e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:49:29.747469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c2d11a-ba34-41be-b32e-92b394e16fe1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:49:29.747985 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c2d11a-ba34-41be-b32e-92b394e16fe1 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:29.748264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c2d11a-ba34-41be-b32e-92b394e16fe1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:29.748533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c2d11a-ba34-41be-b32e-92b394e16fe1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:29.750136 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c1c2d11a-ba34-41be-b32e-92b394e16fe1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:29.869327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c2d11a-ba34-41be-b32e-92b394e16fe1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:49:29.870790 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c2d11a-ba34-41be-b32e-92b394e16fe1 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:49:29.890851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 498/1017] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:49:29 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:49:34.323357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88701875-6e40-4e23-aec3-2497cdcb08b5 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:34.323918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88701875-6e40-4e23-aec3-2497cdcb08b5 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:34.323918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88701875-6e40-4e23-aec3-2497cdcb08b5 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:34.325576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 520/1018] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:49:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:49:34.332537 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bc8734-e1a3-4dfe-bac9-9888db176931 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:34.333340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bc8734-e1a3-4dfe-bac9-9888db176931 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:34.333340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bc8734-e1a3-4dfe-bac9-9888db176931 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:34.340947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:34.419106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-44bc8734-e1a3-4dfe-bac9-9888db176931 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:49:34.420256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-44bc8734-e1a3-4dfe-bac9-9888db176931 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:34.532570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 499/1019] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:49:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:49:34.826897 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88701875-6e40-4e23-aec3-2497cdcb08b5 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:49:34.828453 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-88701875-6e40-4e23-aec3-2497cdcb08b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:34.881475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:34.943693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88701875-6e40-4e23-aec3-2497cdcb08b5 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:49:35.025273 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc6da4f-db95-4bfd-9144-73c2433235c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:49:35.026086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc6da4f-db95-4bfd-9144-73c2433235c1 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:35.026367 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc6da4f-db95-4bfd-9144-73c2433235c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:35.026640 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc6da4f-db95-4bfd-9144-73c2433235c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:35.028333 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-dbc6da4f-db95-4bfd-9144-73c2433235c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:35.131959 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:35.151643 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc6da4f-db95-4bfd-9144-73c2433235c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:49:35.153088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc6da4f-db95-4bfd-9144-73c2433235c1 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:49:35.172860 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 521/1020] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:49:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:49:38.079366 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f788e36-a973-4f5d-a6ca-f55100e2dab9 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:38.079992 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f788e36-a973-4f5d-a6ca-f55100e2dab9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:38.079992 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f788e36-a973-4f5d-a6ca-f55100e2dab9 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:38.081274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 500/1021] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:49:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:49:38.088366 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a60aae-3840-46b7-b19f-495716000939 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:38.088919 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a60aae-3840-46b7-b19f-495716000939 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:38.089022 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a60aae-3840-46b7-b19f-495716000939 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:38.179749 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-89a60aae-3840-46b7-b19f-495716000939 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:49:38.181792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-89a60aae-3840-46b7-b19f-495716000939 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:38.305656 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 522/1022] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:49:38 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:49:38.595430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f788e36-a973-4f5d-a6ca-f55100e2dab9 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:49:38.596999 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6f788e36-a973-4f5d-a6ca-f55100e2dab9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:38.693206 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:38.713861 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f788e36-a973-4f5d-a6ca-f55100e2dab9 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:49:38.792329 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d5e7b3-8cf3-4ce0-a5e3-e471aedcf9e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:49:38.792924 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d5e7b3-8cf3-4ce0-a5e3-e471aedcf9e6 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:38.793197 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d5e7b3-8cf3-4ce0-a5e3-e471aedcf9e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:38.793467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d5e7b3-8cf3-4ce0-a5e3-e471aedcf9e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:38.795086 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c2d5e7b3-8cf3-4ce0-a5e3-e471aedcf9e6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:38.914833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d5e7b3-8cf3-4ce0-a5e3-e471aedcf9e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:49:38.916316 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d5e7b3-8cf3-4ce0-a5e3-e471aedcf9e6 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:49:38.938782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 501/1023] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:49:38 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:49:43.348325 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8da784-5090-46c2-ab38-c1bea2c61f92 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:43.348826 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8da784-5090-46c2-ab38-c1bea2c61f92 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:43.348826 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8da784-5090-46c2-ab38-c1bea2c61f92 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:43.350183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 523/1024] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:49:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:49:43.356071 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fc38d2-332d-439c-9605-304b406e3b08 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:43.356592 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fc38d2-332d-439c-9605-304b406e3b08 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:43.356592 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fc38d2-332d-439c-9605-304b406e3b08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:43.373719 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:43.450510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b1fc38d2-332d-439c-9605-304b406e3b08 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:49:43.452101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b1fc38d2-332d-439c-9605-304b406e3b08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:43.572644 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 502/1025] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:49:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:49:43.858456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8da784-5090-46c2-ab38-c1bea2c61f92 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:49:43.860151 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-3f8da784-5090-46c2-ab38-c1bea2c61f92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:43.927024 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:43.985093 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8da784-5090-46c2-ab38-c1bea2c61f92 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:49:44.066594 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb9ab6d-05fb-4266-bc85-1f19bed29c03 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:49:44.067231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb9ab6d-05fb-4266-bc85-1f19bed29c03 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:44.067508 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb9ab6d-05fb-4266-bc85-1f19bed29c03 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:44.067777 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb9ab6d-05fb-4266-bc85-1f19bed29c03 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:44.069389 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-1bb9ab6d-05fb-4266-bc85-1f19bed29c03 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:44.182392 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:44.188114 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb9ab6d-05fb-4266-bc85-1f19bed29c03 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:49:44.190681 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb9ab6d-05fb-4266-bc85-1f19bed29c03 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:49:44.218295 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 524/1026] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:49:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:49:49.514840 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca2f231-8d29-4c9d-98f1-7d0e3ec6338b None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:49.515852 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca2f231-8d29-4c9d-98f1-7d0e3ec6338b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:49.516334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca2f231-8d29-4c9d-98f1-7d0e3ec6338b None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:49.517981 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 503/1027] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 16 00:49:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:49:49.525811 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7113f9c1-8d30-474a-864f-063538d0f5fb None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:49.526312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7113f9c1-8d30-474a-864f-063538d0f5fb None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:49.526759 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7113f9c1-8d30-474a-864f-063538d0f5fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:49.680409 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-7113f9c1-8d30-474a-864f-063538d0f5fb None None] MFA Rules not processed for user `87785ccd37a642e08a0ad34a4fb1c754`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:49:49.682671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-7113f9c1-8d30-474a-864f-063538d0f5fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:49.964151 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 525/1028] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 16 00:49:49 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 440 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:49:53.570103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=59 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:54.861545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efe698c-efdc-49c0-8cee-b741e15d8c76 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:54.862518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efe698c-efdc-49c0-8cee-b741e15d8c76 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:54.862992 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efe698c-efdc-49c0-8cee-b741e15d8c76 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:54.864581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 504/1029] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 16 00:49:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:49:54.872111 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c614f7a-8174-4eb6-8233-e135b511d6c4 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:54.872640 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c614f7a-8174-4eb6-8233-e135b511d6c4 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:54.873113 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c614f7a-8174-4eb6-8233-e135b511d6c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:54.976804 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:55.022066 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-1c614f7a-8174-4eb6-8233-e135b511d6c4 None None] MFA Rules not processed for user `87785ccd37a642e08a0ad34a4fb1c754`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:49:55.024604 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-1c614f7a-8174-4eb6-8233-e135b511d6c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:55.234882 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 526/1030] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 16 00:49:54 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 365 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:49:56.657796 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d5a0eb-1774-4189-8e7e-5f2fc50b88c9 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:56.658627 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d5a0eb-1774-4189-8e7e-5f2fc50b88c9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:56.658627 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d5a0eb-1774-4189-8e7e-5f2fc50b88c9 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:56.659661 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 505/1031] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 16 00:49:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:49:56.667303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8046f228-ad89-4a59-a0fb-685bf9ccd72e None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:49:56.667841 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8046f228-ad89-4a59-a0fb-685bf9ccd72e None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:49:56.668296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8046f228-ad89-4a59-a0fb-685bf9ccd72e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:49:56.809753 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-8046f228-ad89-4a59-a0fb-685bf9ccd72e None None] MFA Rules not processed for user `87785ccd37a642e08a0ad34a4fb1c754`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:49:56.811144 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8046f228-ad89-4a59-a0fb-685bf9ccd72e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:49:56.892162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:49:56.997033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 527/1032] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 16 00:49:56 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 332 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:50:00.917147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b099ba-36d9-4180-9cd9-db7ec954447e None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:00.917147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b099ba-36d9-4180-9cd9-db7ec954447e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:00.917147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b099ba-36d9-4180-9cd9-db7ec954447e None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:00.918159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 506/1033] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:50:00 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:00.924164 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ed6bf5-d687-46ab-88a8-61becac3f9be None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:00.924670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ed6bf5-d687-46ab-88a8-61becac3f9be None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:00.924751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ed6bf5-d687-46ab-88a8-61becac3f9be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:01.026211 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-e5ed6bf5-d687-46ab-88a8-61becac3f9be None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:01.028183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-e5ed6bf5-d687-46ab-88a8-61becac3f9be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:01.144864 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 528/1034] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:50:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:50:01.453839 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cc1327-ded0-4a43-93a0-b1fb52c51b1e None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:01.454414 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cc1327-ded0-4a43-93a0-b1fb52c51b1e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:01.454693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cc1327-ded0-4a43-93a0-b1fb52c51b1e None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:01.456073 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 507/1035] 10.222.0.25 () {54 vars in 961 bytes} [Sun Oct 16 00:50:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:01.473346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61993e06-f5e1-4737-b300-fb592cdc2c36 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:01.473806 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61993e06-f5e1-4737-b300-fb592cdc2c36 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:01.474183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61993e06-f5e1-4737-b300-fb592cdc2c36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:01.495350 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:01.582373 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-61993e06-f5e1-4737-b300-fb592cdc2c36 None None] MFA Rules not processed for user `4eba26813e154595a12a67fe4be5eeff`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:01.583812 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-61993e06-f5e1-4737-b300-fb592cdc2c36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:01.760195 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 529/1036] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Oct 16 00:50:01 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:50:01.770757 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cc1327-ded0-4a43-93a0-b1fb52c51b1e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:50:01.773868 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d2cc1327-ded0-4a43-93a0-b1fb52c51b1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:01.957703 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cc1327-ded0-4a43-93a0-b1fb52c51b1e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:50:02.043779 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:02.093509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60239323-2e0d-4e1a-b456-df8a05061375 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:50:02.094128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60239323-2e0d-4e1a-b456-df8a05061375 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:02.094405 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60239323-2e0d-4e1a-b456-df8a05061375 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:02.094677 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60239323-2e0d-4e1a-b456-df8a05061375 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:02.096508 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-60239323-2e0d-4e1a-b456-df8a05061375 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:02.222729 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60239323-2e0d-4e1a-b456-df8a05061375 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:50:02.224501 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60239323-2e0d-4e1a-b456-df8a05061375 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:50:02.249290 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 508/1037] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:50:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 479 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:50:04.738763 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2fb30a-63da-4ec4-a221-0b3f637fffb7 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:04.739433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2fb30a-63da-4ec4-a221-0b3f637fffb7 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:04.739433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2fb30a-63da-4ec4-a221-0b3f637fffb7 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:04.746499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9980fe2c-1823-4f21-8a67-2ea7a206d525 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:04.747250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9980fe2c-1823-4f21-8a67-2ea7a206d525 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:04.747250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9980fe2c-1823-4f21-8a67-2ea7a206d525 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:04.751391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 530/1038] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:50:04 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:04.844431 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-9980fe2c-1823-4f21-8a67-2ea7a206d525 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:04.845671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9980fe2c-1823-4f21-8a67-2ea7a206d525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:04.891475 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:04.965507 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 509/1039] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:50:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:50:05.266933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2fb30a-63da-4ec4-a221-0b3f637fffb7 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:50:05.273274 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-cd2fb30a-63da-4ec4-a221-0b3f637fffb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:05.374046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:05.546735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2fb30a-63da-4ec4-a221-0b3f637fffb7 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:50:05.713028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6777df-22fa-43bf-955f-82546b12abf9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:50:05.713832 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6777df-22fa-43bf-955f-82546b12abf9 service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:05.714224 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6777df-22fa-43bf-955f-82546b12abf9 service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:05.714608 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6777df-22fa-43bf-955f-82546b12abf9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:05.716515 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8a6777df-22fa-43bf-955f-82546b12abf9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:05.749455 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:05.865068 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6777df-22fa-43bf-955f-82546b12abf9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:50:05.867386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6777df-22fa-43bf-955f-82546b12abf9 service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:50:05.891434 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 531/1040] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:50:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 626 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:50:08.560095 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10417fe-d372-4a4c-8038-d71cb83f586e None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:08.560893 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10417fe-d372-4a4c-8038-d71cb83f586e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:08.560893 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10417fe-d372-4a4c-8038-d71cb83f586e None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:08.562028 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 510/1041] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:50:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:08.567931 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8903c89-f5d2-4a79-b578-a18bee4edb59 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:08.568670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8903c89-f5d2-4a79-b578-a18bee4edb59 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:08.568670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8903c89-f5d2-4a79-b578-a18bee4edb59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:08.649893 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-d8903c89-f5d2-4a79-b578-a18bee4edb59 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:08.651270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d8903c89-f5d2-4a79-b578-a18bee4edb59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:08.780447 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:08.791550 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 532/1042] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:50:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:50:09.110179 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bbe0e7-91c6-4efa-83ea-6f3340fb5cd3 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:09.110820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bbe0e7-91c6-4efa-83ea-6f3340fb5cd3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:09.111210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bbe0e7-91c6-4efa-83ea-6f3340fb5cd3 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:09.113145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 511/1043] 10.222.0.25 () {54 vars in 961 bytes} [Sun Oct 16 00:50:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:09.119297 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf695f8-c126-4add-a23a-806c5c6ffa10 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:09.119704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf695f8-c126-4add-a23a-806c5c6ffa10 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:09.120065 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf695f8-c126-4add-a23a-806c5c6ffa10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:09.237092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-fbf695f8-c126-4add-a23a-806c5c6ffa10 None None] MFA Rules not processed for user `4eba26813e154595a12a67fe4be5eeff`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:09.239390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-fbf695f8-c126-4add-a23a-806c5c6ffa10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:09.457860 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 533/1044] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Oct 16 00:50:09 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 339 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:50:09.466833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bbe0e7-91c6-4efa-83ea-6f3340fb5cd3 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:50:09.468497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-45bbe0e7-91c6-4efa-83ea-6f3340fb5cd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:09.696677 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bbe0e7-91c6-4efa-83ea-6f3340fb5cd3 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:50:09.726442 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:09.876690 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7e9525-5d2f-4b12-96d7-8aafe7115024 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:50:09.877654 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7e9525-5d2f-4b12-96d7-8aafe7115024 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:09.878093 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7e9525-5d2f-4b12-96d7-8aafe7115024 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:09.878432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7e9525-5d2f-4b12-96d7-8aafe7115024 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:09.880331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8f7e9525-5d2f-4b12-96d7-8aafe7115024 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:10.010584 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7e9525-5d2f-4b12-96d7-8aafe7115024 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:50:10.012527 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7e9525-5d2f-4b12-96d7-8aafe7115024 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:50:10.037478 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 512/1045] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:50:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 571 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:50:10.068417 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75d47cb-77e6-4d89-bd16-5f7d352b6257 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:10.069508 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75d47cb-77e6-4d89-bd16-5f7d352b6257 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:10.069620 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75d47cb-77e6-4d89-bd16-5f7d352b6257 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:10.071835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 534/1046] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:50:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:10.082619 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7044c7-0957-44fa-a7ad-eddf792cc6d0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:10.083133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7044c7-0957-44fa-a7ad-eddf792cc6d0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:10.083516 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7044c7-0957-44fa-a7ad-eddf792cc6d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:10.135144 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:10.197168 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-fb7044c7-0957-44fa-a7ad-eddf792cc6d0 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:10.198843 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fb7044c7-0957-44fa-a7ad-eddf792cc6d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:10.365673 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 513/1047] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:50:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 285 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:50:10.668957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8791d14f-f839-44fc-819c-2d3af0d6e234 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:10.669189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8791d14f-f839-44fc-819c-2d3af0d6e234 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:10.669353 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8791d14f-f839-44fc-819c-2d3af0d6e234 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:10.670924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 535/1048] 10.222.0.25 () {54 vars in 963 bytes} [Sun Oct 16 00:50:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:10.676532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edd0069-bff6-4528-bdeb-66c5279c3645 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:10.676892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edd0069-bff6-4528-bdeb-66c5279c3645 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:10.677090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edd0069-bff6-4528-bdeb-66c5279c3645 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:10.808991 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4edd0069-bff6-4528-bdeb-66c5279c3645 None None] MFA Rules not processed for user `81fa2042309c48e1859b601d8bee1d5b`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:10.812061 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4edd0069-bff6-4528-bdeb-66c5279c3645 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:10.921618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:10.988294 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 514/1049] 10.222.0.25 () {58 vars in 1060 bytes} [Sun Oct 16 00:50:10 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 313 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:50:11.010930 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8791d14f-f839-44fc-819c-2d3af0d6e234 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:50:11.010930 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8791d14f-f839-44fc-819c-2d3af0d6e234 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:11.153092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:11.245869 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8791d14f-f839-44fc-819c-2d3af0d6e234 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:50:11.364166 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd0555b-2380-4546-952f-39dbd7f194d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:50:11.365182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd0555b-2380-4546-952f-39dbd7f194d1 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:11.365660 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd0555b-2380-4546-952f-39dbd7f194d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:11.366662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd0555b-2380-4546-952f-39dbd7f194d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:11.369751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-2fd0555b-2380-4546-952f-39dbd7f194d1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:11.508835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:11.526371 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd0555b-2380-4546-952f-39dbd7f194d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:50:11.528257 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd0555b-2380-4546-952f-39dbd7f194d1 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:50:11.549499 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 536/1050] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:50:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 545 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:50:12.405539 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e8ad84-2bce-418d-8bc9-e47d7246d509 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:12.406223 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e8ad84-2bce-418d-8bc9-e47d7246d509 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:12.407116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e8ad84-2bce-418d-8bc9-e47d7246d509 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:12.411741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 515/1051] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:50:12 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:12.418196 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f787fc-0a9b-4853-99eb-106a8f9e2689 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:12.418713 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f787fc-0a9b-4853-99eb-106a8f9e2689 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:12.418894 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f787fc-0a9b-4853-99eb-106a8f9e2689 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:12.501363 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-83f787fc-0a9b-4853-99eb-106a8f9e2689 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:12.502536 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-83f787fc-0a9b-4853-99eb-106a8f9e2689 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:12.616686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 537/1052] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:50:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:50:12.927855 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e8ad84-2bce-418d-8bc9-e47d7246d509 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:50:12.930009 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-84e8ad84-2bce-418d-8bc9-e47d7246d509 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:13.141977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e8ad84-2bce-418d-8bc9-e47d7246d509 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:50:13.221330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:13.313908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbd1cbd-1576-4e56-8a46-7ac7aca58936 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:50:13.314706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbd1cbd-1576-4e56-8a46-7ac7aca58936 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:13.315054 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbd1cbd-1576-4e56-8a46-7ac7aca58936 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:13.315334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbd1cbd-1576-4e56-8a46-7ac7aca58936 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:13.317067 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-abbd1cbd-1576-4e56-8a46-7ac7aca58936 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:13.438845 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbd1cbd-1576-4e56-8a46-7ac7aca58936 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:50:13.440550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbd1cbd-1576-4e56-8a46-7ac7aca58936 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:50:13.464926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 516/1053] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:50:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 538 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:50:15.823614 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cba14e-e1f8-4754-a590-bfcfdc617e2a None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:15.824250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cba14e-e1f8-4754-a590-bfcfdc617e2a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:15.824250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cba14e-e1f8-4754-a590-bfcfdc617e2a None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:15.825808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 538/1054] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:50:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:15.832072 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd4cb30-3e97-4967-a5e5-a3c46f97d092 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:15.832564 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd4cb30-3e97-4967-a5e5-a3c46f97d092 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:15.832804 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd4cb30-3e97-4967-a5e5-a3c46f97d092 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:15.910073 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-7cd4cb30-3e97-4967-a5e5-a3c46f97d092 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:15.911335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7cd4cb30-3e97-4967-a5e5-a3c46f97d092 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:15.919527 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:16.029890 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 517/1055] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:50:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:50:16.323988 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cba14e-e1f8-4754-a590-bfcfdc617e2a None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:50:16.325935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c5cba14e-e1f8-4754-a590-bfcfdc617e2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:16.395840 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:16.550332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cba14e-e1f8-4754-a590-bfcfdc617e2a None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:50:16.720983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b9e304-e2d7-4744-9e44-d310480e32fa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:50:16.721625 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b9e304-e2d7-4744-9e44-d310480e32fa service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:16.722060 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b9e304-e2d7-4744-9e44-d310480e32fa service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:16.722522 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b9e304-e2d7-4744-9e44-d310480e32fa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:16.725515 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d8b9e304-e2d7-4744-9e44-d310480e32fa service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:16.757232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:16.866878 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b9e304-e2d7-4744-9e44-d310480e32fa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:50:16.868526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b9e304-e2d7-4744-9e44-d310480e32fa service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:50:16.890031 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 539/1056] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:50:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 566 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:50:19.249084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3743e9-fb49-4848-a4cc-f8349581c391 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:19.249630 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3743e9-fb49-4848-a4cc-f8349581c391 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:19.249630 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3743e9-fb49-4848-a4cc-f8349581c391 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:19.251243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 518/1057] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:50:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:19.257477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803b80c9-41a8-4282-b64f-1bc57a0888d8 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:19.258224 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803b80c9-41a8-4282-b64f-1bc57a0888d8 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:19.258224 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803b80c9-41a8-4282-b64f-1bc57a0888d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:19.333963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-803b80c9-41a8-4282-b64f-1bc57a0888d8 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:19.336149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-803b80c9-41a8-4282-b64f-1bc57a0888d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:19.453223 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:19.466905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 540/1058] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:50:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:50:19.767902 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3743e9-fb49-4848-a4cc-f8349581c391 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:50:19.769871 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cb3743e9-fb49-4848-a4cc-f8349581c391 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:19.997165 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:20.012247 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3743e9-fb49-4848-a4cc-f8349581c391 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:50:20.223154 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f9a476-b5f0-4949-a3ba-5c573ed5bc5d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:50:20.224284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f9a476-b5f0-4949-a3ba-5c573ed5bc5d service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:20.224284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f9a476-b5f0-4949-a3ba-5c573ed5bc5d service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:20.225747 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f9a476-b5f0-4949-a3ba-5c573ed5bc5d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:20.226951 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-54f9a476-b5f0-4949-a3ba-5c573ed5bc5d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:20.327490 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f9a476-b5f0-4949-a3ba-5c573ed5bc5d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:50:20.329124 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f9a476-b5f0-4949-a3ba-5c573ed5bc5d service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:50:20.340860 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:20.350686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 519/1059] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:50:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 583 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:50:22.648382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b5550a-6aa7-4845-8167-cd8fc945778d None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:22.649613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b5550a-6aa7-4845-8167-cd8fc945778d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:22.649613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b5550a-6aa7-4845-8167-cd8fc945778d None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:22.651119 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 541/1060] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:50:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:50:22.656918 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9dd27a-647a-4afb-97a7-f5e69dec7e27 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:22.657399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9dd27a-647a-4afb-97a7-f5e69dec7e27 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:22.657520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9dd27a-647a-4afb-97a7-f5e69dec7e27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:22.735592 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4e9dd27a-647a-4afb-97a7-f5e69dec7e27 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:22.736837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4e9dd27a-647a-4afb-97a7-f5e69dec7e27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:22.895937 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 520/1061] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:50:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:50:23.194745 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b5550a-6aa7-4845-8167-cd8fc945778d None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:50:23.196423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-a2b5550a-6aa7-4845-8167-cd8fc945778d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:23.363545 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b5550a-6aa7-4845-8167-cd8fc945778d None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:50:23.444249 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:23.511594 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b1836d-2ecd-4143-80be-b1ef1a1ba8aa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:50:23.512282 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b1836d-2ecd-4143-80be-b1ef1a1ba8aa service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:23.512529 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b1836d-2ecd-4143-80be-b1ef1a1ba8aa service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:23.512805 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b1836d-2ecd-4143-80be-b1ef1a1ba8aa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:23.514443 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-e5b1836d-2ecd-4143-80be-b1ef1a1ba8aa service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:23.625815 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b1836d-2ecd-4143-80be-b1ef1a1ba8aa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:50:23.627403 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b1836d-2ecd-4143-80be-b1ef1a1ba8aa service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:50:23.647731 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 542/1062] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:50:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:50:25.863390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7c591b-6b55-45eb-94ee-804c01813232 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:25.863970 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7c591b-6b55-45eb-94ee-804c01813232 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:25.863970 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7c591b-6b55-45eb-94ee-804c01813232 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:25.865522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 521/1063] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:50:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:25.872352 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fb0c7a-e90b-485f-b925-be122bdc8a1a None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:25.872935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fb0c7a-e90b-485f-b925-be122bdc8a1a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:25.872935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fb0c7a-e90b-485f-b925-be122bdc8a1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:25.943011 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-99fb0c7a-e90b-485f-b925-be122bdc8a1a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:25.944868 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-99fb0c7a-e90b-485f-b925-be122bdc8a1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:25.980932 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:26.046076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 543/1064] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:50:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:50:26.406018 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7c591b-6b55-45eb-94ee-804c01813232 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:50:26.407708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-aa7c591b-6b55-45eb-94ee-804c01813232 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:26.524395 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:26.610574 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7c591b-6b55-45eb-94ee-804c01813232 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:50:26.739756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fb3705-e561-4b2f-b61f-1855bc38ac8d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:50:26.740950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fb3705-e561-4b2f-b61f-1855bc38ac8d service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:26.741441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fb3705-e561-4b2f-b61f-1855bc38ac8d service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:26.741654 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fb3705-e561-4b2f-b61f-1855bc38ac8d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:26.743384 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e1fb3705-e561-4b2f-b61f-1855bc38ac8d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:26.788394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:26.862103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fb3705-e561-4b2f-b61f-1855bc38ac8d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:50:26.863698 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fb3705-e561-4b2f-b61f-1855bc38ac8d service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:50:26.884845 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 522/1065] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:50:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 479 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:50:28.685537 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5777efc0-60c5-4763-99d2-c02e735fe837 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:28.686282 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5777efc0-60c5-4763-99d2-c02e735fe837 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:28.686282 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5777efc0-60c5-4763-99d2-c02e735fe837 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:28.687372 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 544/1066] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:50:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:28.694363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64eb4d87-ed59-44ed-ae19-ec5ae023cbe3 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:28.694891 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64eb4d87-ed59-44ed-ae19-ec5ae023cbe3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:28.695490 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64eb4d87-ed59-44ed-ae19-ec5ae023cbe3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:28.799051 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-64eb4d87-ed59-44ed-ae19-ec5ae023cbe3 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:28.800986 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-64eb4d87-ed59-44ed-ae19-ec5ae023cbe3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:28.915331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 523/1067] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:50:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:50:29.247609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507ed199-fe45-491e-ac83-b76a81e1ca77 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:29.247609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507ed199-fe45-491e-ac83-b76a81e1ca77 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:29.247609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507ed199-fe45-491e-ac83-b76a81e1ca77 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:29.247609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 545/1068] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:50:29 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:29.250109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943fe977-e583-431d-9afb-7f1818ceaeda None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:29.250400 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943fe977-e583-431d-9afb-7f1818ceaeda None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:29.250677 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943fe977-e583-431d-9afb-7f1818ceaeda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:29.267263 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:29.274163 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117ca587-a419-4a51-8ec0-59d3e397915e None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:29.274654 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117ca587-a419-4a51-8ec0-59d3e397915e None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:29.275115 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117ca587-a419-4a51-8ec0-59d3e397915e None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:29.276649 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 546/1069] 10.222.0.25 () {54 vars in 962 bytes} [Sun Oct 16 00:50:29 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:29.283524 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259aee42-6530-48df-b04b-f063e61e5c8b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:29.284045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259aee42-6530-48df-b04b-f063e61e5c8b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:29.284451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259aee42-6530-48df-b04b-f063e61e5c8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:29.332653 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-943fe977-e583-431d-9afb-7f1818ceaeda None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:29.335383 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-943fe977-e583-431d-9afb-7f1818ceaeda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:29.376015 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-259aee42-6530-48df-b04b-f063e61e5c8b None None] MFA Rules not processed for user `78473c29cef449b89bc9185de67f1bf7`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:29.377258 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-259aee42-6530-48df-b04b-f063e61e5c8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:29.465607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 547/1070] 10.222.0.25 () {58 vars in 1059 bytes} [Sun Oct 16 00:50:29 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:50:29.474848 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259aee42-6530-48df-b04b-f063e61e5c8b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:50:29.477690 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 524/1071] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:50:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:50:29.482784 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259aee42-6530-48df-b04b-f063e61e5c8b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:50:29.506792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:29.561677 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3f7b56-a39e-499f-832d-172d38b23df8 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '78473c29cef449b89bc9185de67f1bf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:50:29.562290 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3f7b56-a39e-499f-832d-172d38b23df8 service glance] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:29.562567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3f7b56-a39e-499f-832d-172d38b23df8 service glance] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:29.562833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3f7b56-a39e-499f-832d-172d38b23df8 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:29.564479 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8e3f7b56-a39e-499f-832d-172d38b23df8 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:29.695568 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3f7b56-a39e-499f-832d-172d38b23df8 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:50:29.697266 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3f7b56-a39e-499f-832d-172d38b23df8 service glance] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:50:29.720445 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 548/1072] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:50:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 246 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:50:29.923453 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943fe977-e583-431d-9afb-7f1818ceaeda None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:50:29.924698 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-943fe977-e583-431d-9afb-7f1818ceaeda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:30.038824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:30.074160 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943fe977-e583-431d-9afb-7f1818ceaeda None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:50:30.265505 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c622517f-73a5-4ea0-b49b-9078c8dee602 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:50:30.266647 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c622517f-73a5-4ea0-b49b-9078c8dee602 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:30.267020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c622517f-73a5-4ea0-b49b-9078c8dee602 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:30.267298 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c622517f-73a5-4ea0-b49b-9078c8dee602 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:30.269460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c622517f-73a5-4ea0-b49b-9078c8dee602 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:30.392758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:30.396251 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c622517f-73a5-4ea0-b49b-9078c8dee602 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:50:30.397988 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c622517f-73a5-4ea0-b49b-9078c8dee602 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:50:30.436323 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 525/1073] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:50:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 513 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:50:30.597489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee96097-f578-4ece-92bc-1e1c35ebf4ed None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:30.597830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee96097-f578-4ece-92bc-1e1c35ebf4ed None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:30.598055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee96097-f578-4ece-92bc-1e1c35ebf4ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:30.730590 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-eee96097-f578-4ece-92bc-1e1c35ebf4ed None None] MFA Rules not processed for user `33b136afbf8a4d98a9d62053b68707e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:30.731961 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-eee96097-f578-4ece-92bc-1e1c35ebf4ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:30.734230 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:30.859667 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 549/1074] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 16 00:50:30 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:50:30.922183 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea37dfa-dbda-45b4-af35-9c0433726cb4 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:30.922739 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea37dfa-dbda-45b4-af35-9c0433726cb4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:30.923121 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea37dfa-dbda-45b4-af35-9c0433726cb4 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:30.925912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 526/1075] 10.222.0.25 () {54 vars in 943 bytes} [Sun Oct 16 00:50:30 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:30.942007 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22560cf4-c9f9-4a3b-aa47-69bd48834665 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:30.942468 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22560cf4-c9f9-4a3b-aa47-69bd48834665 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:30.942891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22560cf4-c9f9-4a3b-aa47-69bd48834665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:31.082989 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-22560cf4-c9f9-4a3b-aa47-69bd48834665 None None] MFA Rules not processed for user `de64d498da78448394b0d4d1851a2789`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:31.084521 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-22560cf4-c9f9-4a3b-aa47-69bd48834665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:31.188210 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:31.234147 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 550/1076] 10.222.0.25 () {58 vars in 1040 bytes} [Sun Oct 16 00:50:30 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:50:31.245091 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea37dfa-dbda-45b4-af35-9c0433726cb4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:50:31.247473 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bea37dfa-dbda-45b4-af35-9c0433726cb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:31.415605 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea37dfa-dbda-45b4-af35-9c0433726cb4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:50:31.506976 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:31.553330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2495b8ff-58fb-4951-8790-3a39f9d65574 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de64d498da78448394b0d4d1851a2789', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:50:31.554053 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2495b8ff-58fb-4951-8790-3a39f9d65574 service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:31.554340 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2495b8ff-58fb-4951-8790-3a39f9d65574 service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:31.554617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2495b8ff-58fb-4951-8790-3a39f9d65574 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:31.556485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2495b8ff-58fb-4951-8790-3a39f9d65574 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:31.658722 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2495b8ff-58fb-4951-8790-3a39f9d65574 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:50:31.660407 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2495b8ff-58fb-4951-8790-3a39f9d65574 service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:50:31.670411 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 527/1077] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 16 00:50:31 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 16 00:50:32.894901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07519e15-9d77-4e4a-abba-b5d39dc82f60 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:32.895568 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07519e15-9d77-4e4a-abba-b5d39dc82f60 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:32.895568 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07519e15-9d77-4e4a-abba-b5d39dc82f60 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:32.897855 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 551/1078] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:50:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:32.903672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4263b172-2f44-493d-b974-addc7032b596 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:32.904206 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4263b172-2f44-493d-b974-addc7032b596 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:32.904311 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4263b172-2f44-493d-b974-addc7032b596 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:33.003602 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4263b172-2f44-493d-b974-addc7032b596 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:33.006376 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4263b172-2f44-493d-b974-addc7032b596 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:33.076845 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:33.126494 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 528/1079] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:50:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:50:33.426878 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07519e15-9d77-4e4a-abba-b5d39dc82f60 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:50:33.428988 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-07519e15-9d77-4e4a-abba-b5d39dc82f60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:33.614669 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07519e15-9d77-4e4a-abba-b5d39dc82f60 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:50:33.691404 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:33.766092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1738cc99-87cf-4c95-9d88-024000934656 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:50:33.766719 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1738cc99-87cf-4c95-9d88-024000934656 service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:33.767040 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1738cc99-87cf-4c95-9d88-024000934656 service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:33.767319 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1738cc99-87cf-4c95-9d88-024000934656 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:33.768977 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-1738cc99-87cf-4c95-9d88-024000934656 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:33.890396 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1738cc99-87cf-4c95-9d88-024000934656 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:50:33.892583 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1738cc99-87cf-4c95-9d88-024000934656 service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:50:33.919199 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 552/1080] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:50:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 493 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:50:36.272875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df85ba2-1448-4064-84fd-ad674a54254e None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:36.273498 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df85ba2-1448-4064-84fd-ad674a54254e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:36.273498 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df85ba2-1448-4064-84fd-ad674a54254e None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:36.274897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 529/1081] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:50:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:36.281610 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15898f95-209f-422a-8b1f-fee4e1062aa4 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:36.282167 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15898f95-209f-422a-8b1f-fee4e1062aa4 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:36.282871 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15898f95-209f-422a-8b1f-fee4e1062aa4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:36.369763 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-15898f95-209f-422a-8b1f-fee4e1062aa4 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:36.371020 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-15898f95-209f-422a-8b1f-fee4e1062aa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:36.388515 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:36.508827 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 553/1082] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:50:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:50:37.057821 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df85ba2-1448-4064-84fd-ad674a54254e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:50:37.059960 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0df85ba2-1448-4064-84fd-ad674a54254e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:37.327138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df85ba2-1448-4064-84fd-ad674a54254e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:50:37.395911 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:37.498521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb73f8f-6c03-46f4-bc9a-afe577c01258 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:50:37.499605 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb73f8f-6c03-46f4-bc9a-afe577c01258 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:37.499877 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb73f8f-6c03-46f4-bc9a-afe577c01258 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:37.500240 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb73f8f-6c03-46f4-bc9a-afe577c01258 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:37.501828 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-aeb73f8f-6c03-46f4-bc9a-afe577c01258 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:37.630309 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb73f8f-6c03-46f4-bc9a-afe577c01258 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:50:37.632824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb73f8f-6c03-46f4-bc9a-afe577c01258 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:50:37.661203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 530/1083] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:50:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 604 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:50:40.410069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408378f0-aeef-403b-b045-4d76ef97b293 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:40.410618 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408378f0-aeef-403b-b045-4d76ef97b293 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:40.410618 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408378f0-aeef-403b-b045-4d76ef97b293 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:40.412153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 554/1084] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:50:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:40.419967 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d659c2-1591-4a9c-96a1-183774b3b3b4 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:40.422098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d659c2-1591-4a9c-96a1-183774b3b3b4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:40.422608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d659c2-1591-4a9c-96a1-183774b3b3b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:40.525146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:40.530184 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-24d659c2-1591-4a9c-96a1-183774b3b3b4 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:40.531944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-24d659c2-1591-4a9c-96a1-183774b3b3b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:40.667013 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 531/1085] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:50:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:50:40.973419 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408378f0-aeef-403b-b045-4d76ef97b293 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:50:40.975674 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-408378f0-aeef-403b-b045-4d76ef97b293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:41.172911 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:41.179906 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408378f0-aeef-403b-b045-4d76ef97b293 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:50:41.377635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33ed435-6d83-49ce-a3fe-14f03fba69c6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:50:41.378770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33ed435-6d83-49ce-a3fe-14f03fba69c6 service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:41.379279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33ed435-6d83-49ce-a3fe-14f03fba69c6 service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:41.379592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33ed435-6d83-49ce-a3fe-14f03fba69c6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:41.383410 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-e33ed435-6d83-49ce-a3fe-14f03fba69c6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:41.558149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33ed435-6d83-49ce-a3fe-14f03fba69c6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:50:41.559786 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33ed435-6d83-49ce-a3fe-14f03fba69c6 service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:50:41.578877 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:41.585045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 555/1086] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:50:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 612 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:50:50.661689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=43 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:55.492838 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9367aa-8a83-4ca8-8665-d5b61432e801 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:55.493615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9367aa-8a83-4ca8-8665-d5b61432e801 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:55.493615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9367aa-8a83-4ca8-8665-d5b61432e801 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:55.494757 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 532/1087] 10.222.0.25 () {54 vars in 918 bytes} [Sun Oct 16 00:50:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:55.501247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f27e36-c1a9-47b7-a972-c7b8166fbffb None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:55.501690 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f27e36-c1a9-47b7-a972-c7b8166fbffb None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:55.501794 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f27e36-c1a9-47b7-a972-c7b8166fbffb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:55.595449 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-d1f27e36-c1a9-47b7-a972-c7b8166fbffb None None] MFA Rules not processed for user `87785ccd37a642e08a0ad34a4fb1c754`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:55.596694 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d1f27e36-c1a9-47b7-a972-c7b8166fbffb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:55.716798 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 556/1088] 10.222.0.25 () {58 vars in 1015 bytes} [Sun Oct 16 00:50:55 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:50:55.732052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dadfa9-44c5-41c0-acf6-e23613991aa9 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:55.732335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dadfa9-44c5-41c0-acf6-e23613991aa9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:55.732611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dadfa9-44c5-41c0-acf6-e23613991aa9 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:55.733987 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 533/1089] 10.222.0.25 () {54 vars in 918 bytes} [Sun Oct 16 00:50:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:55.738972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4236c18d-7138-4ae4-9c60-ee50e6f0633d None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:55.739257 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4236c18d-7138-4ae4-9c60-ee50e6f0633d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:55.739541 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4236c18d-7138-4ae4-9c60-ee50e6f0633d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:55.819863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:55.823760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-4236c18d-7138-4ae4-9c60-ee50e6f0633d None None] MFA Rules not processed for user `4eba26813e154595a12a67fe4be5eeff`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:55.825525 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4236c18d-7138-4ae4-9c60-ee50e6f0633d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:55.956487 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 557/1090] 10.222.0.25 () {58 vars in 1015 bytes} [Sun Oct 16 00:50:55 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:50:58.619809 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5faa5c-8cb0-442e-8f1c-7d1101d25563 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:58.620392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5faa5c-8cb0-442e-8f1c-7d1101d25563 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:58.620392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5faa5c-8cb0-442e-8f1c-7d1101d25563 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:58.621762 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 534/1091] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 16 00:50:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:50:58.628054 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd477eaf-343f-4d11-b763-708b03b426fb None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:58.628631 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd477eaf-343f-4d11-b763-708b03b426fb None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:58.628631 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd477eaf-343f-4d11-b763-708b03b426fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:58.700526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-dd477eaf-343f-4d11-b763-708b03b426fb None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:50:58.701923 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-dd477eaf-343f-4d11-b763-708b03b426fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:58.758500 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:58.803557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 558/1092] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 16 00:50:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:50:59.098013 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5faa5c-8cb0-442e-8f1c-7d1101d25563 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:50:59.099661 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5c5faa5c-8cb0-442e-8f1c-7d1101d25563 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:59.250333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5faa5c-8cb0-442e-8f1c-7d1101d25563 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:50:59.324274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:50:59.367310 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57171412-ef42-4ba8-a2df-5975e5c2e62d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:50:59.367833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57171412-ef42-4ba8-a2df-5975e5c2e62d service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:50:59.368114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57171412-ef42-4ba8-a2df-5975e5c2e62d service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:50:59.368389 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57171412-ef42-4ba8-a2df-5975e5c2e62d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:50:59.370010 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-57171412-ef42-4ba8-a2df-5975e5c2e62d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:50:59.463662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57171412-ef42-4ba8-a2df-5975e5c2e62d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:50:59.465294 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57171412-ef42-4ba8-a2df-5975e5c2e62d service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:50:59.483696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 535/1093] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:50:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:00.665674 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aad4078-0e16-4a2c-851b-8810aa0f6847 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:00.666246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aad4078-0e16-4a2c-851b-8810aa0f6847 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:00.666246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aad4078-0e16-4a2c-851b-8810aa0f6847 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:00.667616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 559/1094] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 16 00:51:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:00.674266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104dc042-af74-47e0-b16b-b170d669c5d5 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:00.674747 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104dc042-af74-47e0-b16b-b170d669c5d5 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:00.674837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104dc042-af74-47e0-b16b-b170d669c5d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:00.756151 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-104dc042-af74-47e0-b16b-b170d669c5d5 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:00.757399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-104dc042-af74-47e0-b16b-b170d669c5d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:00.790467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:00.871000 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 536/1095] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 16 00:51:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:51:01.026379 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aad4078-0e16-4a2c-851b-8810aa0f6847 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:01.027922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-7aad4078-0e16-4a2c-851b-8810aa0f6847 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:01.192805 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aad4078-0e16-4a2c-851b-8810aa0f6847 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:01.240835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:01.322698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d00252-b9fb-4f7a-bc4b-ef4f90e07d5c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:01.323346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d00252-b9fb-4f7a-bc4b-ef4f90e07d5c service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:01.323629 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d00252-b9fb-4f7a-bc4b-ef4f90e07d5c service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:01.323903 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d00252-b9fb-4f7a-bc4b-ef4f90e07d5c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:01.325547 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-a9d00252-b9fb-4f7a-bc4b-ef4f90e07d5c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:01.429845 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d00252-b9fb-4f7a-bc4b-ef4f90e07d5c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:01.430507 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d00252-b9fb-4f7a-bc4b-ef4f90e07d5c service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:01.451309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 560/1096] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:02.608289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b504cdd9-1a2f-4ae0-bcab-2e7c6a88e8d0 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:02.608865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b504cdd9-1a2f-4ae0-bcab-2e7c6a88e8d0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:02.608865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b504cdd9-1a2f-4ae0-bcab-2e7c6a88e8d0 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:02.610230 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 537/1097] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 16 00:51:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:02.617533 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ac0352-5ceb-49d4-b211-7475c60b07be None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:02.618093 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ac0352-5ceb-49d4-b211-7475c60b07be None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:02.618093 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ac0352-5ceb-49d4-b211-7475c60b07be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:02.673759 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:02.697044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-53ac0352-5ceb-49d4-b211-7475c60b07be None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:02.698184 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-53ac0352-5ceb-49d4-b211-7475c60b07be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:02.798069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 561/1098] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 16 00:51:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:51:02.953355 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b504cdd9-1a2f-4ae0-bcab-2e7c6a88e8d0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:02.954803 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b504cdd9-1a2f-4ae0-bcab-2e7c6a88e8d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:03.104338 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b504cdd9-1a2f-4ae0-bcab-2e7c6a88e8d0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:03.117443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:03.221755 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc70a1a-9c40-42d0-b84d-a5d2ff9dd8e8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:03.222276 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc70a1a-9c40-42d0-b84d-a5d2ff9dd8e8 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:03.222550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc70a1a-9c40-42d0-b84d-a5d2ff9dd8e8 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:03.222817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc70a1a-9c40-42d0-b84d-a5d2ff9dd8e8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:03.224947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5dc70a1a-9c40-42d0-b84d-a5d2ff9dd8e8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:03.315358 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc70a1a-9c40-42d0-b84d-a5d2ff9dd8e8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:03.316968 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc70a1a-9c40-42d0-b84d-a5d2ff9dd8e8 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:03.335646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 538/1099] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:04.472102 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28770a3e-4715-4977-9402-8972d4ba7209 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:04.472692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28770a3e-4715-4977-9402-8972d4ba7209 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:04.472692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28770a3e-4715-4977-9402-8972d4ba7209 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:04.474172 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 562/1100] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 16 00:51:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:04.481169 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d6bedc-f585-4c5c-ab5d-79ba60f69054 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:04.481722 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d6bedc-f585-4c5c-ab5d-79ba60f69054 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:04.481722 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d6bedc-f585-4c5c-ab5d-79ba60f69054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:04.502325 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:04.568743 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a9d6bedc-f585-4c5c-ab5d-79ba60f69054 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:04.570018 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a9d6bedc-f585-4c5c-ab5d-79ba60f69054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:04.680946 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 539/1101] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 16 00:51:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:51:04.831609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28770a3e-4715-4977-9402-8972d4ba7209 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:04.833765 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-28770a3e-4715-4977-9402-8972d4ba7209 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:04.948101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:04.984846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28770a3e-4715-4977-9402-8972d4ba7209 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:05.101894 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7022ccc2-360f-446b-9ce9-b35ae921322e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:05.102509 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7022ccc2-360f-446b-9ce9-b35ae921322e service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:05.102795 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7022ccc2-360f-446b-9ce9-b35ae921322e service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:05.103103 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7022ccc2-360f-446b-9ce9-b35ae921322e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:05.104752 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-7022ccc2-360f-446b-9ce9-b35ae921322e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:05.184430 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:05.199604 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7022ccc2-360f-446b-9ce9-b35ae921322e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:05.201404 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7022ccc2-360f-446b-9ce9-b35ae921322e service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:05.224606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 563/1102] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:06.357678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dde430-bbb0-4b09-8307-b123216a005a None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:06.358259 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dde430-bbb0-4b09-8307-b123216a005a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:06.358259 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22dde430-bbb0-4b09-8307-b123216a005a None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:06.359672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 540/1103] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 16 00:51:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:06.366828 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec0aa86-5819-4a2b-ba14-5e92736af880 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:06.367386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec0aa86-5819-4a2b-ba14-5e92736af880 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:06.367494 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec0aa86-5819-4a2b-ba14-5e92736af880 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:06.449531 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-2ec0aa86-5819-4a2b-ba14-5e92736af880 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:06.450801 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-2ec0aa86-5819-4a2b-ba14-5e92736af880 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:06.559506 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 564/1104] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 16 00:51:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:51:06.712657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22dde430-bbb0-4b09-8307-b123216a005a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:06.714084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-22dde430-bbb0-4b09-8307-b123216a005a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:06.799696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:06.867382 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22dde430-bbb0-4b09-8307-b123216a005a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:06.984153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa879dc-ac50-4ff8-9438-9fdd96cec216 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:06.984696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa879dc-ac50-4ff8-9438-9fdd96cec216 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:06.984976 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa879dc-ac50-4ff8-9438-9fdd96cec216 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:06.985251 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa879dc-ac50-4ff8-9438-9fdd96cec216 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:06.986882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8fa879dc-ac50-4ff8-9438-9fdd96cec216 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:07.039199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:07.081772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa879dc-ac50-4ff8-9438-9fdd96cec216 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:07.083985 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa879dc-ac50-4ff8-9438-9fdd96cec216 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:07.102338 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 541/1105] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:08.175435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d4a531-264e-4fcd-86bb-5d88b4d72a24 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:08.176025 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d4a531-264e-4fcd-86bb-5d88b4d72a24 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:08.176025 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d4a531-264e-4fcd-86bb-5d88b4d72a24 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:08.178035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 565/1106] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 16 00:51:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:08.184716 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec5c4de-cc7a-44f9-ba58-c36e271db751 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:08.185236 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec5c4de-cc7a-44f9-ba58-c36e271db751 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:08.185354 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec5c4de-cc7a-44f9-ba58-c36e271db751 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:08.265283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-0ec5c4de-cc7a-44f9-ba58-c36e271db751 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:08.266522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0ec5c4de-cc7a-44f9-ba58-c36e271db751 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:08.369704 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 542/1107] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 16 00:51:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:51:08.520390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d4a531-264e-4fcd-86bb-5d88b4d72a24 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:08.522100 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-42d4a531-264e-4fcd-86bb-5d88b4d72a24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:08.573265 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:08.670967 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d4a531-264e-4fcd-86bb-5d88b4d72a24 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:08.785231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85bfb122-23bc-434a-9e54-e9ea70a59b36 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:08.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bfb122-23bc-434a-9e54-e9ea70a59b36 service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:08.786037 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bfb122-23bc-434a-9e54-e9ea70a59b36 service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:08.786809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85bfb122-23bc-434a-9e54-e9ea70a59b36 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:08.788550 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-85bfb122-23bc-434a-9e54-e9ea70a59b36 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:08.796200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:08.884216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bfb122-23bc-434a-9e54-e9ea70a59b36 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:08.885812 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85bfb122-23bc-434a-9e54-e9ea70a59b36 service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:08.906223 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 566/1108] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:09.980252 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1df3650-c558-489d-b119-9df299ada5d7 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:09.980779 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1df3650-c558-489d-b119-9df299ada5d7 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:09.980779 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1df3650-c558-489d-b119-9df299ada5d7 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:09.982068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 543/1109] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 16 00:51:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:09.988792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1e7357-9fda-432b-a554-5e01ecc7d2f3 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:09.990197 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1e7357-9fda-432b-a554-5e01ecc7d2f3 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:09.990197 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1e7357-9fda-432b-a554-5e01ecc7d2f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:10.070468 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-4b1e7357-9fda-432b-a554-5e01ecc7d2f3 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:10.071741 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4b1e7357-9fda-432b-a554-5e01ecc7d2f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:10.148512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:10.179887 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 567/1110] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 16 00:51:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:51:10.331889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1df3650-c558-489d-b119-9df299ada5d7 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:10.333985 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f1df3650-c558-489d-b119-9df299ada5d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:10.352213 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:10.486188 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1df3650-c558-489d-b119-9df299ada5d7 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:10.574841 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:10.602408 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddbae7d-b19f-48f5-a67c-64f84dcddc96 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:10.602969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddbae7d-b19f-48f5-a67c-64f84dcddc96 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:10.603252 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddbae7d-b19f-48f5-a67c-64f84dcddc96 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:10.603522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddbae7d-b19f-48f5-a67c-64f84dcddc96 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:10.605160 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9ddbae7d-b19f-48f5-a67c-64f84dcddc96 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:10.696630 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddbae7d-b19f-48f5-a67c-64f84dcddc96 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:10.698250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddbae7d-b19f-48f5-a67c-64f84dcddc96 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:10.717938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 544/1111] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:11.794827 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55793a1c-a020-4fdf-8dc1-29f5fba56f9b None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:11.795432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55793a1c-a020-4fdf-8dc1-29f5fba56f9b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:11.795530 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55793a1c-a020-4fdf-8dc1-29f5fba56f9b None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:11.796971 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 568/1112] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 16 00:51:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:11.804428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd90635f-374a-462a-8638-18d7275e47fe None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:11.804961 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd90635f-374a-462a-8638-18d7275e47fe None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:11.804961 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd90635f-374a-462a-8638-18d7275e47fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:11.894581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-fd90635f-374a-462a-8638-18d7275e47fe None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:11.896436 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fd90635f-374a-462a-8638-18d7275e47fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:11.940264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:12.001807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 545/1113] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 16 00:51:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:51:12.155835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55793a1c-a020-4fdf-8dc1-29f5fba56f9b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:12.157376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-55793a1c-a020-4fdf-8dc1-29f5fba56f9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:12.309071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55793a1c-a020-4fdf-8dc1-29f5fba56f9b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:12.365941 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:12.432216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc871b43-dc0d-4096-aa02-3b06144b4400 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:12.432856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc871b43-dc0d-4096-aa02-3b06144b4400 service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:12.433139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc871b43-dc0d-4096-aa02-3b06144b4400 service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:12.433437 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc871b43-dc0d-4096-aa02-3b06144b4400 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:12.435112 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-cc871b43-dc0d-4096-aa02-3b06144b4400 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:12.526348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc871b43-dc0d-4096-aa02-3b06144b4400 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:12.527968 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc871b43-dc0d-4096-aa02-3b06144b4400 service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:12.546686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 569/1114] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:13.621109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0a3f74-1f1e-4854-bf5f-77207e7681f4 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:13.621688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0a3f74-1f1e-4854-bf5f-77207e7681f4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:13.621688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0a3f74-1f1e-4854-bf5f-77207e7681f4 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:13.623046 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 546/1115] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 16 00:51:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:13.629872 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaa8f4e-c197-4251-bbe0-6d8dece0c743 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:13.630904 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaa8f4e-c197-4251-bbe0-6d8dece0c743 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:13.630904 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaa8f4e-c197-4251-bbe0-6d8dece0c743 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:13.713624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-ebaa8f4e-c197-4251-bbe0-6d8dece0c743 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:13.714878 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-ebaa8f4e-c197-4251-bbe0-6d8dece0c743 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:13.717038 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:13.821268 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 570/1116] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 16 00:51:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:51:13.976206 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0a3f74-1f1e-4854-bf5f-77207e7681f4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:13.977671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fa0a3f74-1f1e-4854-bf5f-77207e7681f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:14.127326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0a3f74-1f1e-4854-bf5f-77207e7681f4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:14.159623 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:14.249635 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf2a087-de96-4447-b8ff-606819c749c4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:14.250161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf2a087-de96-4447-b8ff-606819c749c4 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:14.250443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf2a087-de96-4447-b8ff-606819c749c4 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:14.250715 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf2a087-de96-4447-b8ff-606819c749c4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:14.252361 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fbf2a087-de96-4447-b8ff-606819c749c4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:14.344773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf2a087-de96-4447-b8ff-606819c749c4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:14.346386 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf2a087-de96-4447-b8ff-606819c749c4 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:14.364907 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 547/1117] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:16.475807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30637c76-a781-4dad-9f62-f8e8aee27a4a None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:16.476343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30637c76-a781-4dad-9f62-f8e8aee27a4a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:16.476343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30637c76-a781-4dad-9f62-f8e8aee27a4a None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:16.477711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 571/1118] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 16 00:51:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:16.483291 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44138a92-01ff-454a-a7c9-0714a9ba2480 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:16.483754 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44138a92-01ff-454a-a7c9-0714a9ba2480 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:16.483854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44138a92-01ff-454a-a7c9-0714a9ba2480 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:16.526541 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:16.554787 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-44138a92-01ff-454a-a7c9-0714a9ba2480 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:16.556084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-44138a92-01ff-454a-a7c9-0714a9ba2480 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:16.654508 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 548/1119] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 16 00:51:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:51:16.945742 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30637c76-a781-4dad-9f62-f8e8aee27a4a None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:16.947349 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-30637c76-a781-4dad-9f62-f8e8aee27a4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:17.054600 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30637c76-a781-4dad-9f62-f8e8aee27a4a None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:17.088950 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:17.131432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572d2636-6ced-4eb8-8c3b-3863f1910517 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:17.132038 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572d2636-6ced-4eb8-8c3b-3863f1910517 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:17.132318 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572d2636-6ced-4eb8-8c3b-3863f1910517 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:17.132579 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572d2636-6ced-4eb8-8c3b-3863f1910517 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:17.134186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-572d2636-6ced-4eb8-8c3b-3863f1910517 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:17.247767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572d2636-6ced-4eb8-8c3b-3863f1910517 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:17.249490 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572d2636-6ced-4eb8-8c3b-3863f1910517 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:17.267899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 572/1120] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:19.662178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65f3731-a835-42c2-b700-716fbfdd5712 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:19.662749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65f3731-a835-42c2-b700-716fbfdd5712 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:19.662749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65f3731-a835-42c2-b700-716fbfdd5712 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:19.664212 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 549/1121] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 16 00:51:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:19.669752 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b682a39-d36d-4cae-8e83-71cd9431d96c None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:19.670180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b682a39-d36d-4cae-8e83-71cd9431d96c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:19.670285 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b682a39-d36d-4cae-8e83-71cd9431d96c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:19.743320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-5b682a39-d36d-4cae-8e83-71cd9431d96c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:19.744600 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-5b682a39-d36d-4cae-8e83-71cd9431d96c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:19.758670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:19.844095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 573/1122] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 16 00:51:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:51:20.128088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65f3731-a835-42c2-b700-716fbfdd5712 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:20.129661 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d65f3731-a835-42c2-b700-716fbfdd5712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:20.245332 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65f3731-a835-42c2-b700-716fbfdd5712 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:20.258766 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:20.320216 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72a7737-2a34-4a28-b6ab-4e64bc45e930 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:20.320824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72a7737-2a34-4a28-b6ab-4e64bc45e930 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:20.321109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72a7737-2a34-4a28-b6ab-4e64bc45e930 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:20.321389 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72a7737-2a34-4a28-b6ab-4e64bc45e930 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:20.323035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b72a7737-2a34-4a28-b6ab-4e64bc45e930 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:20.434071 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72a7737-2a34-4a28-b6ab-4e64bc45e930 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:20.435561 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72a7737-2a34-4a28-b6ab-4e64bc45e930 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:20.453623 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 550/1123] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:22.970996 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e116bddf-9f84-48f3-9a82-b1a29d3ec669 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:22.971607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e116bddf-9f84-48f3-9a82-b1a29d3ec669 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:22.971607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e116bddf-9f84-48f3-9a82-b1a29d3ec669 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:22.972891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 574/1124] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 16 00:51:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:22.979028 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb6e35-eb91-4f2c-9736-177ba53456ce None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:22.979528 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb6e35-eb91-4f2c-9736-177ba53456ce None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:22.979668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb6e35-eb91-4f2c-9736-177ba53456ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:23.029764 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:23.049158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-cdcb6e35-eb91-4f2c-9736-177ba53456ce None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:23.050371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cdcb6e35-eb91-4f2c-9736-177ba53456ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:23.162892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 551/1125] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 16 00:51:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:51:23.455790 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e116bddf-9f84-48f3-9a82-b1a29d3ec669 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:23.457333 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-e116bddf-9f84-48f3-9a82-b1a29d3ec669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:23.569984 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e116bddf-9f84-48f3-9a82-b1a29d3ec669 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:23.611823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:23.648759 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187fbec0-a9a9-475f-9529-119072d3b837 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:23.649375 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187fbec0-a9a9-475f-9529-119072d3b837 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:23.649658 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187fbec0-a9a9-475f-9529-119072d3b837 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:23.649933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187fbec0-a9a9-475f-9529-119072d3b837 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:23.651586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-187fbec0-a9a9-475f-9529-119072d3b837 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:23.764187 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187fbec0-a9a9-475f-9529-119072d3b837 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:23.765639 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187fbec0-a9a9-475f-9529-119072d3b837 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:23.784947 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 575/1126] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:26.115485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9b514-2e89-452f-b267-d7be9fac99f9 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:26.116035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9b514-2e89-452f-b267-d7be9fac99f9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:26.116035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f9b514-2e89-452f-b267-d7be9fac99f9 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:26.117398 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 552/1127] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 16 00:51:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:26.122960 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f163aff-8e0b-423f-bfd4-777d219dfe20 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:26.123393 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f163aff-8e0b-423f-bfd4-777d219dfe20 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:26.123519 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f163aff-8e0b-423f-bfd4-777d219dfe20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:26.198769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-6f163aff-8e0b-423f-bfd4-777d219dfe20 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:26.200033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6f163aff-8e0b-423f-bfd4-777d219dfe20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:26.223937 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:26.307986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 576/1128] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 16 00:51:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:51:26.602726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f9b514-2e89-452f-b267-d7be9fac99f9 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:26.604295 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-53f9b514-2e89-452f-b267-d7be9fac99f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:26.718658 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f9b514-2e89-452f-b267-d7be9fac99f9 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:26.743069 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:26.805353 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22cc37c-609c-4a05-9b7c-4403a50b7480 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:26.805967 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22cc37c-609c-4a05-9b7c-4403a50b7480 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:26.806242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22cc37c-609c-4a05-9b7c-4403a50b7480 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:26.806509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22cc37c-609c-4a05-9b7c-4403a50b7480 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:26.808180 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b22cc37c-609c-4a05-9b7c-4403a50b7480 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:26.924360 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22cc37c-609c-4a05-9b7c-4403a50b7480 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:26.925816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22cc37c-609c-4a05-9b7c-4403a50b7480 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:26.944241 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 553/1129] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:29.269192 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45aead5a-07d3-40e1-b330-b388b30c31d0 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:29.269871 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45aead5a-07d3-40e1-b330-b388b30c31d0 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:29.269871 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45aead5a-07d3-40e1-b330-b388b30c31d0 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:29.271666 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 577/1130] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 16 00:51:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:29.277259 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd02a4a7-3c0c-4b9c-a36f-27a5bf5221cd None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:29.277758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd02a4a7-3c0c-4b9c-a36f-27a5bf5221cd None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:29.277866 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd02a4a7-3c0c-4b9c-a36f-27a5bf5221cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:29.349874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:29.352912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-cd02a4a7-3c0c-4b9c-a36f-27a5bf5221cd None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:29.354767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cd02a4a7-3c0c-4b9c-a36f-27a5bf5221cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:29.457481 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 554/1131] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 16 00:51:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:51:29.742322 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45aead5a-07d3-40e1-b330-b388b30c31d0 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:29.743906 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-45aead5a-07d3-40e1-b330-b388b30c31d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:29.850648 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45aead5a-07d3-40e1-b330-b388b30c31d0 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:29.914674 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:29.940101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b7cb1c-0229-4ee3-9046-1b7ff7af4795 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:29.940721 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b7cb1c-0229-4ee3-9046-1b7ff7af4795 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:29.940997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b7cb1c-0229-4ee3-9046-1b7ff7af4795 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:29.941267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b7cb1c-0229-4ee3-9046-1b7ff7af4795 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:29.943572 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-33b7cb1c-0229-4ee3-9046-1b7ff7af4795 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:30.054147 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b7cb1c-0229-4ee3-9046-1b7ff7af4795 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:30.055631 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b7cb1c-0229-4ee3-9046-1b7ff7af4795 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:30.074177 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 578/1132] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:32.381497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a90421c-cbc5-40f1-921a-7bd19615f53b None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:32.382157 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a90421c-cbc5-40f1-921a-7bd19615f53b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:32.383078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a90421c-cbc5-40f1-921a-7bd19615f53b None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:32.384907 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 555/1133] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 16 00:51:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:32.391424 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78345db0-bc1a-4db6-b447-191c606bca95 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:32.392010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78345db0-bc1a-4db6-b447-191c606bca95 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:32.392010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78345db0-bc1a-4db6-b447-191c606bca95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:32.474633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-78345db0-bc1a-4db6-b447-191c606bca95 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:32.476098 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-78345db0-bc1a-4db6-b447-191c606bca95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:32.511854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:32.601028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 579/1134] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 16 00:51:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:51:32.927638 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a90421c-cbc5-40f1-921a-7bd19615f53b None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:32.929196 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6a90421c-cbc5-40f1-921a-7bd19615f53b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:33.044643 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a90421c-cbc5-40f1-921a-7bd19615f53b None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:33.075222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:33.123660 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a24727-b1e0-401c-af99-525e2a4fc885 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:33.124273 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a24727-b1e0-401c-af99-525e2a4fc885 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:33.124557 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a24727-b1e0-401c-af99-525e2a4fc885 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:33.124825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a24727-b1e0-401c-af99-525e2a4fc885 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:33.126454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-67a24727-b1e0-401c-af99-525e2a4fc885 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:33.243453 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a24727-b1e0-401c-af99-525e2a4fc885 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:33.244860 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a24727-b1e0-401c-af99-525e2a4fc885 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:33.264318 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 556/1135] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:51:35.535127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f4a635-29bf-435b-b865-baf89195ff74 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:35.535822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f4a635-29bf-435b-b865-baf89195ff74 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:35.535822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f4a635-29bf-435b-b865-baf89195ff74 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:35.537060 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 580/1136] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 16 00:51:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:35.542554 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407a0b92-4db8-4765-9e19-15b98673ac8b None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:35.543096 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407a0b92-4db8-4765-9e19-15b98673ac8b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:35.543201 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407a0b92-4db8-4765-9e19-15b98673ac8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:35.616575 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-407a0b92-4db8-4765-9e19-15b98673ac8b None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:35.617792 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-407a0b92-4db8-4765-9e19-15b98673ac8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:35.626339 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:35.721327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 557/1137] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 16 00:51:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:51:36.004963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f4a635-29bf-435b-b865-baf89195ff74 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:36.006570 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-62f4a635-29bf-435b-b865-baf89195ff74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:36.114180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f4a635-29bf-435b-b865-baf89195ff74 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:36.173221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:36.192680 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52ac2d1-b697-401d-b74e-3ba144bbc8bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:36.193308 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52ac2d1-b697-401d-b74e-3ba144bbc8bf service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:36.193597 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52ac2d1-b697-401d-b74e-3ba144bbc8bf service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:36.193850 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52ac2d1-b697-401d-b74e-3ba144bbc8bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:36.195547 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-f52ac2d1-b697-401d-b74e-3ba144bbc8bf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:36.308393 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52ac2d1-b697-401d-b74e-3ba144bbc8bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:36.309847 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52ac2d1-b697-401d-b74e-3ba144bbc8bf service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:36.330401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 581/1138] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:38.555423 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137a239c-18ab-4bf8-9b53-fc6091756e2e None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:38.556010 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137a239c-18ab-4bf8-9b53-fc6091756e2e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:38.556010 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137a239c-18ab-4bf8-9b53-fc6091756e2e None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:38.557322 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 558/1139] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 16 00:51:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:38.563000 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30275d1-9a5c-4982-b85c-1549afbcf875 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:38.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30275d1-9a5c-4982-b85c-1549afbcf875 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:38.563580 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30275d1-9a5c-4982-b85c-1549afbcf875 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:38.635015 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-d30275d1-9a5c-4982-b85c-1549afbcf875 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:38.636273 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d30275d1-9a5c-4982-b85c-1549afbcf875 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:38.677254 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:38.738324 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 582/1140] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 16 00:51:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:51:39.027185 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137a239c-18ab-4bf8-9b53-fc6091756e2e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:39.028751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-137a239c-18ab-4bf8-9b53-fc6091756e2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:39.137711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137a239c-18ab-4bf8-9b53-fc6091756e2e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:39.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:39.220348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a239ed8d-ba00-4669-9a72-4c7e0755478b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:39.220959 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a239ed8d-ba00-4669-9a72-4c7e0755478b service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:39.221868 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a239ed8d-ba00-4669-9a72-4c7e0755478b service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:39.222161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a239ed8d-ba00-4669-9a72-4c7e0755478b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:39.224660 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a239ed8d-ba00-4669-9a72-4c7e0755478b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:39.336478 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a239ed8d-ba00-4669-9a72-4c7e0755478b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:39.337939 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a239ed8d-ba00-4669-9a72-4c7e0755478b service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:39.357271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 559/1141] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:41.551851 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af423e6a-0491-4cce-91cc-ed901b0ef6e6 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:41.552435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af423e6a-0491-4cce-91cc-ed901b0ef6e6 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:41.552559 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af423e6a-0491-4cce-91cc-ed901b0ef6e6 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:41.553831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 583/1142] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 16 00:51:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:41.559435 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581be231-667b-4b5f-be8e-4c7cdaad3005 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:41.559958 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581be231-667b-4b5f-be8e-4c7cdaad3005 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:41.559958 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581be231-667b-4b5f-be8e-4c7cdaad3005 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:41.627859 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-581be231-667b-4b5f-be8e-4c7cdaad3005 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:41.629152 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-581be231-667b-4b5f-be8e-4c7cdaad3005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:41.648200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:41.731980 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 560/1143] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 16 00:51:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:51:42.018803 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af423e6a-0491-4cce-91cc-ed901b0ef6e6 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:42.020419 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-af423e6a-0491-4cce-91cc-ed901b0ef6e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:42.129082 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af423e6a-0491-4cce-91cc-ed901b0ef6e6 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:42.199067 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:42.206674 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6923329f-fa4e-4970-bf5c-177f1cb9cf58 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:42.207371 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6923329f-fa4e-4970-bf5c-177f1cb9cf58 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:42.207829 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6923329f-fa4e-4970-bf5c-177f1cb9cf58 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:42.208310 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6923329f-fa4e-4970-bf5c-177f1cb9cf58 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:42.209957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6923329f-fa4e-4970-bf5c-177f1cb9cf58 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:42.324173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6923329f-fa4e-4970-bf5c-177f1cb9cf58 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:42.325622 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6923329f-fa4e-4970-bf5c-177f1cb9cf58 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:42.344171 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 584/1144] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:44.551480 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038fde07-0e53-457e-ab79-90247ab3e4f3 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:44.552073 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038fde07-0e53-457e-ab79-90247ab3e4f3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:44.552073 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038fde07-0e53-457e-ab79-90247ab3e4f3 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:44.553392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 561/1145] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 16 00:51:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:51:44.559322 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbfdd62-a1f4-48a1-99fc-85e731964709 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:44.559860 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbfdd62-a1f4-48a1-99fc-85e731964709 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:44.559860 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbfdd62-a1f4-48a1-99fc-85e731964709 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:44.628895 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-7fbfdd62-a1f4-48a1-99fc-85e731964709 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:51:44.630632 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-7fbfdd62-a1f4-48a1-99fc-85e731964709 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:44.681870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:44.733483 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 585/1146] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 16 00:51:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:51:45.026236 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038fde07-0e53-457e-ab79-90247ab3e4f3 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:51:45.027836 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-038fde07-0e53-457e-ab79-90247ab3e4f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:45.142176 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038fde07-0e53-457e-ab79-90247ab3e4f3 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:51:45.192286 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:45.225317 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d4ae11-d163-41fd-8e1e-cdbc1529d75e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:51:45.225926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d4ae11-d163-41fd-8e1e-cdbc1529d75e service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:51:45.226207 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d4ae11-d163-41fd-8e1e-cdbc1529d75e service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:51:45.226477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d4ae11-d163-41fd-8e1e-cdbc1529d75e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:51:45.228594 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-30d4ae11-d163-41fd-8e1e-cdbc1529d75e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:51:45.341792 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d4ae11-d163-41fd-8e1e-cdbc1529d75e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:51:45.343290 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d4ae11-d163-41fd-8e1e-cdbc1529d75e service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:51:45.362229 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 562/1147] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:51:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:51:54.731983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=22 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:51:55.359868 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=66 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:52:05.581511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b14f731-c296-4d53-8540-60821bad3c9e None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:05.582441 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b14f731-c296-4d53-8540-60821bad3c9e None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:05.582441 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b14f731-c296-4d53-8540-60821bad3c9e None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:05.583531 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 586/1148] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:52:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:52:05.588965 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2446cfeb-47b6-422f-953e-64bc1f91ffe8 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:05.589403 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2446cfeb-47b6-422f-953e-64bc1f91ffe8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:05.589568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2446cfeb-47b6-422f-953e-64bc1f91ffe8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:05.655902 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2446cfeb-47b6-422f-953e-64bc1f91ffe8 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:52:05.657119 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2446cfeb-47b6-422f-953e-64bc1f91ffe8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:05.756815 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 563/1149] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:52:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:52:06.061508 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6ee6f2-27db-4dd0-9911-95ec91dd1ecd None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:06.061797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6ee6f2-27db-4dd0-9911-95ec91dd1ecd None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:06.062076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6ee6f2-27db-4dd0-9911-95ec91dd1ecd None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:06.063455 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 587/1150] 10.222.0.25 () {54 vars in 962 bytes} [Sun Oct 16 00:52:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:52:06.068849 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87473204-251f-41eb-8bd2-75871ec8cd53 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:06.069137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87473204-251f-41eb-8bd2-75871ec8cd53 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:06.069415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87473204-251f-41eb-8bd2-75871ec8cd53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:06.147401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-87473204-251f-41eb-8bd2-75871ec8cd53 None None] MFA Rules not processed for user `78473c29cef449b89bc9185de67f1bf7`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:52:06.148655 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-87473204-251f-41eb-8bd2-75871ec8cd53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:06.170651 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:52:06.234215 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 564/1151] 10.222.0.25 () {58 vars in 1059 bytes} [Sun Oct 16 00:52:06 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:52:06.242451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6ee6f2-27db-4dd0-9911-95ec91dd1ecd None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:52:06.244203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-7a6ee6f2-27db-4dd0-9911-95ec91dd1ecd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:06.350382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6ee6f2-27db-4dd0-9911-95ec91dd1ecd None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:52:06.427337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d6eb6e-b6c6-4096-a5aa-a3f687cbbaa2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '78473c29cef449b89bc9185de67f1bf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:52:06.428034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d6eb6e-b6c6-4096-a5aa-a3f687cbbaa2 service glance] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:06.428230 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d6eb6e-b6c6-4096-a5aa-a3f687cbbaa2 service glance] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:06.428496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d6eb6e-b6c6-4096-a5aa-a3f687cbbaa2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:06.430123 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-f2d6eb6e-b6c6-4096-a5aa-a3f687cbbaa2 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:06.486869 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:52:06.543069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d6eb6e-b6c6-4096-a5aa-a3f687cbbaa2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:52:06.544471 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d6eb6e-b6c6-4096-a5aa-a3f687cbbaa2 service glance] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:52:06.570803 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 588/1152] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:52:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:52:08.863743 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06102b4-668b-4bc8-bf04-3f66a1be4976 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:08.864354 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06102b4-668b-4bc8-bf04-3f66a1be4976 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:08.864354 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06102b4-668b-4bc8-bf04-3f66a1be4976 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:08.865723 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 565/1153] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:52:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:52:08.871739 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a307818-a333-497d-b74d-38f25f7e2427 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:08.872245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a307818-a333-497d-b74d-38f25f7e2427 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:08.872366 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a307818-a333-497d-b74d-38f25f7e2427 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:08.947745 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-4a307818-a333-497d-b74d-38f25f7e2427 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:52:08.948982 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4a307818-a333-497d-b74d-38f25f7e2427 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:09.054294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 589/1154] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:52:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:52:09.329739 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06102b4-668b-4bc8-bf04-3f66a1be4976 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:52:09.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f06102b4-668b-4bc8-bf04-3f66a1be4976 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:09.461948 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06102b4-668b-4bc8-bf04-3f66a1be4976 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:52:09.512369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:52:09.557543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b31c12c-df67-433a-932a-e99a94808f8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:52:09.558156 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b31c12c-df67-433a-932a-e99a94808f8d admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:09.558435 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b31c12c-df67-433a-932a-e99a94808f8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:09.558706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b31c12c-df67-433a-932a-e99a94808f8d admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:09.559398 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b31c12c-df67-433a-932a-e99a94808f8d admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:52:09.560503 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: WARNING py.warnings [None req-6b31c12c-df67-433a-932a-e99a94808f8d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 16 00:52:09.560503 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: warnings.warn(msg) Oct 16 00:52:09.560503 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: Oct 16 00:52:09.561646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b31c12c-df67-433a-932a-e99a94808f8d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:52:09.581704 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 566/1155] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 16 00:52:09 2022] GET /identity/v3/projects => generated 2553 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:52:11.826192 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fdb483-35ef-4423-8956-6465cd0c3322 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:11.827022 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fdb483-35ef-4423-8956-6465cd0c3322 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:11.827022 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fdb483-35ef-4423-8956-6465cd0c3322 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:11.828392 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 590/1156] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:52:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:52:11.835126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89830a10-daa7-4916-bc21-e32705abde5f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:11.835629 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89830a10-daa7-4916-bc21-e32705abde5f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:11.835741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89830a10-daa7-4916-bc21-e32705abde5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:11.905314 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-89830a10-daa7-4916-bc21-e32705abde5f None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:52:11.906548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-89830a10-daa7-4916-bc21-e32705abde5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:12.007696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 567/1157] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:52:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:52:12.309377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fdb483-35ef-4423-8956-6465cd0c3322 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:52:12.311033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d5fdb483-35ef-4423-8956-6465cd0c3322 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:12.321936 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:52:12.485757 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fdb483-35ef-4423-8956-6465cd0c3322 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:52:12.568052 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:52:12.601934 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a3e2ad-c565-4b24-a3a2-e085a8b4beae service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:52:12.602558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a3e2ad-c565-4b24-a3a2-e085a8b4beae service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:12.602891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a3e2ad-c565-4b24-a3a2-e085a8b4beae service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:12.603174 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a3e2ad-c565-4b24-a3a2-e085a8b4beae service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:12.604938 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-69a3e2ad-c565-4b24-a3a2-e085a8b4beae service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:12.698092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a3e2ad-c565-4b24-a3a2-e085a8b4beae service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:52:12.699775 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a3e2ad-c565-4b24-a3a2-e085a8b4beae service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:52:12.718574 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 591/1158] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:52:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:52:15.087903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a762c4c6-dc86-4cff-85a4-71ba96a2e04f None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:15.088504 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a762c4c6-dc86-4cff-85a4-71ba96a2e04f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:15.088504 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a762c4c6-dc86-4cff-85a4-71ba96a2e04f None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:15.089815 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 568/1159] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:52:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:52:15.095369 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4661039-cfb1-4e39-934a-92fc5dc2ac06 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:15.095845 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4661039-cfb1-4e39-934a-92fc5dc2ac06 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:15.095943 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4661039-cfb1-4e39-934a-92fc5dc2ac06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:15.170309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-f4661039-cfb1-4e39-934a-92fc5dc2ac06 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:52:15.171569 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-f4661039-cfb1-4e39-934a-92fc5dc2ac06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:15.225402 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:52:15.281491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 592/1160] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:52:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:52:15.587173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a762c4c6-dc86-4cff-85a4-71ba96a2e04f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:52:15.588748 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a762c4c6-dc86-4cff-85a4-71ba96a2e04f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:15.613274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:52:15.696254 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a762c4c6-dc86-4cff-85a4-71ba96a2e04f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:52:15.776288 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a67ba9-de9c-4a3d-b157-a53cbd12d3e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:52:15.777209 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a67ba9-de9c-4a3d-b157-a53cbd12d3e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:15.777649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a67ba9-de9c-4a3d-b157-a53cbd12d3e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:15.777922 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a67ba9-de9c-4a3d-b157-a53cbd12d3e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:15.779694 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a2a67ba9-de9c-4a3d-b157-a53cbd12d3e8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:15.856046 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:52:15.893077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a67ba9-de9c-4a3d-b157-a53cbd12d3e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:52:15.895266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a67ba9-de9c-4a3d-b157-a53cbd12d3e8 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:52:15.914797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 569/1161] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:52:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:52:18.160999 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d4ad96-ee03-44b6-bf8c-88b0586b7a60 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:18.161577 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d4ad96-ee03-44b6-bf8c-88b0586b7a60 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:18.161577 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d4ad96-ee03-44b6-bf8c-88b0586b7a60 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:18.162929 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 593/1162] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:52:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:52:18.168607 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8ab05f-c86b-4e80-9b40-c72f5f57adcf None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:18.169093 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8ab05f-c86b-4e80-9b40-c72f5f57adcf None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:18.169193 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8ab05f-c86b-4e80-9b40-c72f5f57adcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:18.239325 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ab8ab05f-c86b-4e80-9b40-c72f5f57adcf None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:52:18.240556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ab8ab05f-c86b-4e80-9b40-c72f5f57adcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:18.342064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 570/1163] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:52:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:52:18.632341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d4ad96-ee03-44b6-bf8c-88b0586b7a60 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:52:18.633900 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-51d4ad96-ee03-44b6-bf8c-88b0586b7a60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:18.740460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d4ad96-ee03-44b6-bf8c-88b0586b7a60 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:52:18.815467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:52:18.818151 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1c401d-8811-476e-b1f3-7ad1cc7db475 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:52:18.818828 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1c401d-8811-476e-b1f3-7ad1cc7db475 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:18.819396 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1c401d-8811-476e-b1f3-7ad1cc7db475 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:18.819828 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1c401d-8811-476e-b1f3-7ad1cc7db475 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:18.821671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6e1c401d-8811-476e-b1f3-7ad1cc7db475 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:18.932838 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1c401d-8811-476e-b1f3-7ad1cc7db475 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:52:18.934268 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1c401d-8811-476e-b1f3-7ad1cc7db475 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:52:18.952872 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 594/1164] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:52:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:52:21.271458 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f270e0-32cc-4411-98d8-43a9f16d77d5 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:21.272008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f270e0-32cc-4411-98d8-43a9f16d77d5 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:21.272008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f270e0-32cc-4411-98d8-43a9f16d77d5 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:21.273434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 571/1165] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:52:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:52:21.279573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ff8d80-a7ec-46c5-94c0-416bd2cd9958 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:21.280259 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ff8d80-a7ec-46c5-94c0-416bd2cd9958 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:21.280259 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ff8d80-a7ec-46c5-94c0-416bd2cd9958 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:21.367108 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-37ff8d80-a7ec-46c5-94c0-416bd2cd9958 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:52:21.368357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-37ff8d80-a7ec-46c5-94c0-416bd2cd9958 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:21.427832 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:52:21.473958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 595/1166] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:52:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:52:21.773912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f270e0-32cc-4411-98d8-43a9f16d77d5 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:52:21.775460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c5f270e0-32cc-4411-98d8-43a9f16d77d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:21.811810 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:52:21.885082 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f270e0-32cc-4411-98d8-43a9f16d77d5 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:52:21.960832 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67546391-9671-431c-a5e7-a1e56e824b1c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:52:21.961440 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67546391-9671-431c-a5e7-a1e56e824b1c service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:21.961722 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67546391-9671-431c-a5e7-a1e56e824b1c service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:21.961987 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67546391-9671-431c-a5e7-a1e56e824b1c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:21.963625 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-67546391-9671-431c-a5e7-a1e56e824b1c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:22.047673 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:52:22.078274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67546391-9671-431c-a5e7-a1e56e824b1c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:52:22.079758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67546391-9671-431c-a5e7-a1e56e824b1c service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:52:22.098794 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 572/1167] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:52:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:52:24.587675 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ba367a-8c05-41a6-b3b5-19262ffa3644 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:24.588242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ba367a-8c05-41a6-b3b5-19262ffa3644 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:24.588242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ba367a-8c05-41a6-b3b5-19262ffa3644 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:24.589598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 596/1168] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 16 00:52:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:52:24.595473 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094fb822-a28c-4469-a880-ecb2e9b8e87a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:24.595938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094fb822-a28c-4469-a880-ecb2e9b8e87a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:24.596055 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094fb822-a28c-4469-a880-ecb2e9b8e87a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:24.671047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-094fb822-a28c-4469-a880-ecb2e9b8e87a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:52:24.672292 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-094fb822-a28c-4469-a880-ecb2e9b8e87a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:24.781922 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 573/1169] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 16 00:52:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:52:25.072756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ba367a-8c05-41a6-b3b5-19262ffa3644 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:52:25.074301 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d9ba367a-8c05-41a6-b3b5-19262ffa3644 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:25.180986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ba367a-8c05-41a6-b3b5-19262ffa3644 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:52:25.258388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7944a6f7-eaca-4be4-a5b1-39d6b5727155 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:52:25.259023 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7944a6f7-eaca-4be4-a5b1-39d6b5727155 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:52:25.259305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7944a6f7-eaca-4be4-a5b1-39d6b5727155 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:52:25.259577 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7944a6f7-eaca-4be4-a5b1-39d6b5727155 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:52:25.261182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-7944a6f7-eaca-4be4-a5b1-39d6b5727155 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:52:25.271001 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:52:25.378843 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7944a6f7-eaca-4be4-a5b1-39d6b5727155 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:52:25.380237 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7944a6f7-eaca-4be4-a5b1-39d6b5727155 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:52:25.398984 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 597/1170] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:52:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:52:34.780214 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=82 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:52:35.396625 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=50 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:53:40.392226 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2028db75-d4d4-4af0-8526-261b5f4992a4 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:53:40.392226 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2028db75-d4d4-4af0-8526-261b5f4992a4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:53:40.392881 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2028db75-d4d4-4af0-8526-261b5f4992a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:53:40.470914 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2028db75-d4d4-4af0-8526-261b5f4992a4 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:53:40.472135 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2028db75-d4d4-4af0-8526-261b5f4992a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:53:40.567865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 574/1171] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:53:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:53:40.575979 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab3c416-9672-496a-a403-0011e3781718 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:53:40.576502 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab3c416-9672-496a-a403-0011e3781718 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:53:40.576502 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab3c416-9672-496a-a403-0011e3781718 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:53:40.654767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-cab3c416-9672-496a-a403-0011e3781718 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:53:40.655912 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-cab3c416-9672-496a-a403-0011e3781718 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:53:40.754326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 598/1172] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:53:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:53:40.759076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2028db75-d4d4-4af0-8526-261b5f4992a4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:53:40.760602 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2028db75-d4d4-4af0-8526-261b5f4992a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:53:40.771967 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2028db75-d4d4-4af0-8526-261b5f4992a4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:53:40.837269 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:53:40.871770 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442cd893-96b9-4e35-bbbb-c1937d2ce0f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:53:40.872383 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442cd893-96b9-4e35-bbbb-c1937d2ce0f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:53:40.872662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442cd893-96b9-4e35-bbbb-c1937d2ce0f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:53:40.872933 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442cd893-96b9-4e35-bbbb-c1937d2ce0f9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:53:40.873576 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442cd893-96b9-4e35-bbbb-c1937d2ce0f9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:53:40.873881 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442cd893-96b9-4e35-bbbb-c1937d2ce0f9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:53:40.874916 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: WARNING py.warnings [None req-442cd893-96b9-4e35-bbbb-c1937d2ce0f9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 16 00:53:40.874916 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: warnings.warn(msg) Oct 16 00:53:40.874916 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: Oct 16 00:53:40.876431 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442cd893-96b9-4e35-bbbb-c1937d2ce0f9 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:53:40.887601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 575/1173] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:53:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:53:40.897316 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab3c416-9672-496a-a403-0011e3781718 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:53:40.906643 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab3c416-9672-496a-a403-0011e3781718 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:53:40.972783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:53:41.009151 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96dbb89f-42d9-4ca7-b0a5-e9e9632f168f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:53:41.009766 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dbb89f-42d9-4ca7-b0a5-e9e9632f168f admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:53:41.010100 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dbb89f-42d9-4ca7-b0a5-e9e9632f168f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:53:41.010325 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dbb89f-42d9-4ca7-b0a5-e9e9632f168f admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:53:41.010995 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96dbb89f-42d9-4ca7-b0a5-e9e9632f168f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:53:41.012110 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: WARNING py.warnings [None req-96dbb89f-42d9-4ca7-b0a5-e9e9632f168f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 16 00:53:41.012110 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: warnings.warn(msg) Oct 16 00:53:41.012110 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: Oct 16 00:53:41.013352 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96dbb89f-42d9-4ca7-b0a5-e9e9632f168f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:53:41.091259 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 599/1174] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:53:40 2022] POST /identity/v3/projects => generated 379 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:53:41.096069 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442cd893-96b9-4e35-bbbb-c1937d2ce0f9 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:53:41.097594 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-442cd893-96b9-4e35-bbbb-c1937d2ce0f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:53:41.131928 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442cd893-96b9-4e35-bbbb-c1937d2ce0f9 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:53:41.260953 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d84e6af-bf5f-419e-9230-0c041ec773f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:53:41.261570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d84e6af-bf5f-419e-9230-0c041ec773f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:53:41.261849 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d84e6af-bf5f-419e-9230-0c041ec773f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:53:41.262129 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d84e6af-bf5f-419e-9230-0c041ec773f7 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:53:41.262755 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d84e6af-bf5f-419e-9230-0c041ec773f7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:53:41.263804 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: WARNING py.warnings [None req-5d84e6af-bf5f-419e-9230-0c041ec773f7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 16 00:53:41.263804 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: warnings.warn(msg) Oct 16 00:53:41.263804 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: Oct 16 00:53:41.265065 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d84e6af-bf5f-419e-9230-0c041ec773f7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:53:41.321685 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 576/1175] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:53:41 2022] POST /identity/v3/users => generated 290 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:53:41.326000 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96dbb89f-42d9-4ca7-b0a5-e9e9632f168f admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:53:41.327892 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-96dbb89f-42d9-4ca7-b0a5-e9e9632f168f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:53:41.391792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96dbb89f-42d9-4ca7-b0a5-e9e9632f168f admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:53:41.478819 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:53:41.490132 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed918edf-8875-4322-9f67-2e115ba252b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:53:41.490898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed918edf-8875-4322-9f67-2e115ba252b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:53:41.491186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed918edf-8875-4322-9f67-2e115ba252b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:53:41.491450 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed918edf-8875-4322-9f67-2e115ba252b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:53:41.492108 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed918edf-8875-4322-9f67-2e115ba252b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:53:41.493688 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed918edf-8875-4322-9f67-2e115ba252b3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:53:41.506381 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 600/1176] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:53:41 2022] GET /identity/v3/roles => generated 1420 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:53:41.510397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d84e6af-bf5f-419e-9230-0c041ec773f7 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:53:41.511957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5d84e6af-bf5f-419e-9230-0c041ec773f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:53:41.523763 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d84e6af-bf5f-419e-9230-0c041ec773f7 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:53:41.549060 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:53:41.623009 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f16ab1-ed78-4eb3-b323-f9e19f73762e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:53:41.623671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f16ab1-ed78-4eb3-b323-f9e19f73762e admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:53:41.623939 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f16ab1-ed78-4eb3-b323-f9e19f73762e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:53:41.624215 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f16ab1-ed78-4eb3-b323-f9e19f73762e admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:53:41.624864 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f16ab1-ed78-4eb3-b323-f9e19f73762e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:53:41.626513 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f16ab1-ed78-4eb3-b323-f9e19f73762e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:53:41.638992 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 577/1177] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:53:41 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:53:41.643775 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed918edf-8875-4322-9f67-2e115ba252b3 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:53:41.645651 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-ed918edf-8875-4322-9f67-2e115ba252b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:53:41.682994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed918edf-8875-4322-9f67-2e115ba252b3 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:53:41.778699 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20687328-4d9b-490a-8760-c6dbde974390 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:53:41.779408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20687328-4d9b-490a-8760-c6dbde974390 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:53:41.779709 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20687328-4d9b-490a-8760-c6dbde974390 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:53:41.779998 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20687328-4d9b-490a-8760-c6dbde974390 admin admin] PATH_INFO: `/v3/projects/d58f0bf90fb74d0fb8fce1e76d532c13/users/e7235a4d9b29414ebf583fbbd261a876/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:53:41.804500 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20687328-4d9b-490a-8760-c6dbde974390 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d58f0bf90fb74d0fb8fce1e76d532c13, user_id=e7235a4d9b29414ebf583fbbd261a876, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:53:41.807222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20687328-4d9b-490a-8760-c6dbde974390 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:53:41.834473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 601/1178] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:53:41 2022] PUT /identity/v3/projects/d58f0bf90fb74d0fb8fce1e76d532c13/users/e7235a4d9b29414ebf583fbbd261a876/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:53:41.845729 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb21784d-cb9a-4eea-ab2c-c8649081dacc None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:53:41.846009 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb21784d-cb9a-4eea-ab2c-c8649081dacc None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:53:41.846285 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb21784d-cb9a-4eea-ab2c-c8649081dacc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:53:41.905631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-fb21784d-cb9a-4eea-ab2c-c8649081dacc None None] MFA Rules not processed for user `e7235a4d9b29414ebf583fbbd261a876`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:53:41.906966 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fb21784d-cb9a-4eea-ab2c-c8649081dacc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:53:41.983952 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 578/1179] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:53:41 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:53:41.997277 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7f3ccf-131b-4581-98ab-c187cf81bf2c None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:53:41.997582 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7f3ccf-131b-4581-98ab-c187cf81bf2c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:53:41.997891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7f3ccf-131b-4581-98ab-c187cf81bf2c None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:53:41.999272 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 602/1180] 10.222.0.25 () {54 vars in 963 bytes} [Sun Oct 16 00:53:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:53:42.004572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5b5623-51f0-4595-931f-5d3709e31a5d None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:53:42.004864 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5b5623-51f0-4595-931f-5d3709e31a5d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:53:42.005142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5b5623-51f0-4595-931f-5d3709e31a5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:53:42.038364 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:53:42.083899 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4c5b5623-51f0-4595-931f-5d3709e31a5d None None] MFA Rules not processed for user `81fa2042309c48e1859b601d8bee1d5b`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:53:42.085126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4c5b5623-51f0-4595-931f-5d3709e31a5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:53:42.160700 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 579/1181] 10.222.0.25 () {58 vars in 1060 bytes} [Sun Oct 16 00:53:42 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:53:42.170317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7f3ccf-131b-4581-98ab-c187cf81bf2c None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:53:42.171884 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-0d7f3ccf-131b-4581-98ab-c187cf81bf2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:53:42.226019 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:53:42.282277 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7f3ccf-131b-4581-98ab-c187cf81bf2c None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:53:42.357831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8293f446-c56d-4498-be19-d6ab30c1bb9a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:53:42.358440 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8293f446-c56d-4498-be19-d6ab30c1bb9a service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:53:42.358718 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8293f446-c56d-4498-be19-d6ab30c1bb9a service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:53:42.359016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8293f446-c56d-4498-be19-d6ab30c1bb9a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:53:42.361291 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8293f446-c56d-4498-be19-d6ab30c1bb9a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:53:42.462665 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8293f446-c56d-4498-be19-d6ab30c1bb9a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:53:42.464150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8293f446-c56d-4498-be19-d6ab30c1bb9a service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:53:42.467743 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:53:42.482508 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 603/1182] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:53:42 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:53:42.528880 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9066dcae-7856-42ff-90a8-663f196fd98f None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:53:42.529321 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9066dcae-7856-42ff-90a8-663f196fd98f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:53:42.529431 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9066dcae-7856-42ff-90a8-663f196fd98f None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:53:42.530665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 580/1183] 10.222.0.25 () {52 vars in 830 bytes} [Sun Oct 16 00:53:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:53:42.534355 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8293f446-c56d-4498-be19-d6ab30c1bb9a service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:53:42.535586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8293f446-c56d-4498-be19-d6ab30c1bb9a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:53:42.634494 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8293f446-c56d-4498-be19-d6ab30c1bb9a service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:53:42.727641 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67545335-cded-46a1-9c56-6aa67245f7db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:53:42.728338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67545335-cded-46a1-9c56-6aa67245f7db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:53:42.728545 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67545335-cded-46a1-9c56-6aa67245f7db admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:53:42.728813 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67545335-cded-46a1-9c56-6aa67245f7db admin admin] PATH_INFO: `/v3/users/e7235a4d9b29414ebf583fbbd261a876` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:53:42.742992 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67545335-cded-46a1-9c56-6aa67245f7db admin admin] RBAC: Authorizing `identity:delete_user(user_id=e7235a4d9b29414ebf583fbbd261a876)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:53:42.744015 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: WARNING py.warnings [None req-67545335-cded-46a1-9c56-6aa67245f7db admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 16 00:53:42.744015 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: warnings.warn(msg) Oct 16 00:53:42.744015 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: Oct 16 00:53:42.745245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67545335-cded-46a1-9c56-6aa67245f7db admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:53:42.755295 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:53:42.812141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-67545335-cded-46a1-9c56-6aa67245f7db admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e7235a4d9b29414ebf583fbbd261a876'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:53:42.812721 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-67545335-cded-46a1-9c56-6aa67245f7db admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e7235a4d9b29414ebf583fbbd261a876'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:53:42.827566 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-67545335-cded-46a1-9c56-6aa67245f7db admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e7235a4d9b29414ebf583fbbd261a876'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:53:42.856264 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-67545335-cded-46a1-9c56-6aa67245f7db admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e7235a4d9b29414ebf583fbbd261a876'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:53:42.872468 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-67545335-cded-46a1-9c56-6aa67245f7db admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e7235a4d9b29414ebf583fbbd261a876'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:53:42.879763 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 604/1184] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:53:42 2022] DELETE /identity/v3/users/e7235a4d9b29414ebf583fbbd261a876 => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:53:42.884308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c23d6bb-f4c3-4de6-8fba-ebe63c34347e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:53:42.884585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c23d6bb-f4c3-4de6-8fba-ebe63c34347e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:53:42.884861 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c23d6bb-f4c3-4de6-8fba-ebe63c34347e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:53:42.949580 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5c23d6bb-f4c3-4de6-8fba-ebe63c34347e None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:53:42.950844 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5c23d6bb-f4c3-4de6-8fba-ebe63c34347e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:53:43.027259 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:53:43.040919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 581/1185] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:53:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:53:43.051735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67545335-cded-46a1-9c56-6aa67245f7db admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:53:43.052985 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-67545335-cded-46a1-9c56-6aa67245f7db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:53:43.157493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67545335-cded-46a1-9c56-6aa67245f7db admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:53:43.214393 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:53:43.237183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82685634-27d0-4387-851c-69d65b45ff87 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:53:43.237863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82685634-27d0-4387-851c-69d65b45ff87 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:53:43.238145 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82685634-27d0-4387-851c-69d65b45ff87 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:53:43.238421 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82685634-27d0-4387-851c-69d65b45ff87 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:53:43.240079 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-82685634-27d0-4387-851c-69d65b45ff87 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:53:43.351588 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82685634-27d0-4387-851c-69d65b45ff87 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:53:43.353031 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82685634-27d0-4387-851c-69d65b45ff87 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:53:43.371120 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 605/1186] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:53:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:53:43.731339 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c23d6bb-f4c3-4de6-8fba-ebe63c34347e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:53:43.732585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5c23d6bb-f4c3-4de6-8fba-ebe63c34347e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:53:43.745321 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c23d6bb-f4c3-4de6-8fba-ebe63c34347e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:53:43.850804 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc7b192-cb4d-4151-ae74-1bd30df2079b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:53:43.851500 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc7b192-cb4d-4151-ae74-1bd30df2079b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:53:43.851783 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc7b192-cb4d-4151-ae74-1bd30df2079b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:53:43.852060 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc7b192-cb4d-4151-ae74-1bd30df2079b admin admin] PATH_INFO: `/v3/projects/d58f0bf90fb74d0fb8fce1e76d532c13` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:53:43.863963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc7b192-cb4d-4151-ae74-1bd30df2079b admin admin] RBAC: Authorizing `identity:delete_project(project_id=d58f0bf90fb74d0fb8fce1e76d532c13)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:53:43.865029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: WARNING py.warnings [None req-7bc7b192-cb4d-4151-ae74-1bd30df2079b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 16 00:53:43.865029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: warnings.warn(msg) Oct 16 00:53:43.865029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: Oct 16 00:53:43.867067 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc7b192-cb4d-4151-ae74-1bd30df2079b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:53:43.928639 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7bc7b192-cb4d-4151-ae74-1bd30df2079b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd58f0bf90fb74d0fb8fce1e76d532c13', 'request_id': 'req-7bc7b192-cb4d-4151-ae74-1bd30df2079b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:53:43.940835 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7bc7b192-cb4d-4151-ae74-1bd30df2079b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd58f0bf90fb74d0fb8fce1e76d532c13', 'request_id': 'req-7bc7b192-cb4d-4151-ae74-1bd30df2079b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:53:43.955785 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7bc7b192-cb4d-4151-ae74-1bd30df2079b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd58f0bf90fb74d0fb8fce1e76d532c13', 'request_id': 'req-7bc7b192-cb4d-4151-ae74-1bd30df2079b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:53:43.979228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7bc7b192-cb4d-4151-ae74-1bd30df2079b admin admin] The token cache is being invalidate because project d58f0bf90fb74d0fb8fce1e76d532c13 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:53:43.979448 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7bc7b192-cb4d-4151-ae74-1bd30df2079b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:53:43.979847 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7bc7b192-cb4d-4151-ae74-1bd30df2079b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:53:43.981265 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 582/1187] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:53:43 2022] DELETE /identity/v3/projects/d58f0bf90fb74d0fb8fce1e76d532c13 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:53:53.368779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=62 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:53:53.976967 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:55:11.578852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b340548-0a89-43c5-bd50-88a9b0061bed None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:11.579617 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b340548-0a89-43c5-bd50-88a9b0061bed None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:11.579617 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b340548-0a89-43c5-bd50-88a9b0061bed None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:11.580776 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 606/1188] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 16 00:55:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:55:11.585869 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf645fa0-ffa9-496a-8f2b-cdf7cfbbe72b None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:11.586325 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf645fa0-ffa9-496a-8f2b-cdf7cfbbe72b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:11.586441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf645fa0-ffa9-496a-8f2b-cdf7cfbbe72b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:11.669280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-cf645fa0-ffa9-496a-8f2b-cdf7cfbbe72b None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:55:11.670386 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cf645fa0-ffa9-496a-8f2b-cdf7cfbbe72b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:55:11.776873 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 583/1189] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 16 00:55:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:55:11.930781 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b340548-0a89-43c5-bd50-88a9b0061bed None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:55:11.932982 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4b340548-0a89-43c5-bd50-88a9b0061bed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:55:12.053985 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b340548-0a89-43c5-bd50-88a9b0061bed None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:55:12.059374 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5dd586-c9ef-4fed-a3c5-aef957a54bb9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:55:12.059981 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5dd586-c9ef-4fed-a3c5-aef957a54bb9 service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:12.060255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5dd586-c9ef-4fed-a3c5-aef957a54bb9 service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:12.060523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5dd586-c9ef-4fed-a3c5-aef957a54bb9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:12.068706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5dd586-c9ef-4fed-a3c5-aef957a54bb9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:55:12.070296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5dd586-c9ef-4fed-a3c5-aef957a54bb9 service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:55:12.079764 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 607/1190] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:55:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:55:13.184693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454dfcfd-df4a-4488-9064-3f421d9f68da None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:13.185215 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454dfcfd-df4a-4488-9064-3f421d9f68da None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:13.185215 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454dfcfd-df4a-4488-9064-3f421d9f68da None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:13.186534 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 584/1191] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 16 00:55:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:55:13.193627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d162213d-d15b-4a20-8eae-f405175b4d34 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:13.194106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d162213d-d15b-4a20-8eae-f405175b4d34 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:13.194217 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d162213d-d15b-4a20-8eae-f405175b4d34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:13.214510 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-d162213d-d15b-4a20-8eae-f405175b4d34 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:55:13.215775 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d162213d-d15b-4a20-8eae-f405175b4d34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:55:13.223965 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 608/1192] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 16 00:55:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:55:13.377397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454dfcfd-df4a-4488-9064-3f421d9f68da None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:55:13.392199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454dfcfd-df4a-4488-9064-3f421d9f68da None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:55:13.397785 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cbbdb1-f0e1-408b-a5b9-99e7dbcc2263 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:55:13.398389 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cbbdb1-f0e1-408b-a5b9-99e7dbcc2263 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:13.398674 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cbbdb1-f0e1-408b-a5b9-99e7dbcc2263 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:13.398979 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cbbdb1-f0e1-408b-a5b9-99e7dbcc2263 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:13.408097 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3cbbdb1-f0e1-408b-a5b9-99e7dbcc2263 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:55:13.409705 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3cbbdb1-f0e1-408b-a5b9-99e7dbcc2263 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:55:13.419732 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 585/1193] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:55:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:55:13.502160 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3322e7e3-f97c-4c2f-8ce5-76496ddb2d76 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:13.502462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3322e7e3-f97c-4c2f-8ce5-76496ddb2d76 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:13.502744 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3322e7e3-f97c-4c2f-8ce5-76496ddb2d76 None None] PATH_INFO: `/v3` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:13.504378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 609/1194] 10.222.0.36 () {54 vars in 925 bytes} [Sun Oct 16 00:55:13 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:55:13.521541 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10527eaa-5436-4986-8cbe-7c9a67f7a909 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:13.521847 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10527eaa-5436-4986-8cbe-7c9a67f7a909 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:13.522122 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10527eaa-5436-4986-8cbe-7c9a67f7a909 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:13.586150 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-10527eaa-5436-4986-8cbe-7c9a67f7a909 None None] MFA Rules not processed for user `87785ccd37a642e08a0ad34a4fb1c754`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:55:13.587383 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-10527eaa-5436-4986-8cbe-7c9a67f7a909 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:55:13.605438 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:55:13.661156 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 586/1195] 10.222.0.36 () {58 vars in 1015 bytes} [Sun Oct 16 00:55:13 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:55:17.281643 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f87960c-ce54-4d9f-807a-9efeb1ae0c9c None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:17.282180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f87960c-ce54-4d9f-807a-9efeb1ae0c9c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:17.282180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f87960c-ce54-4d9f-807a-9efeb1ae0c9c None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:17.283674 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 610/1196] 10.222.0.25 () {54 vars in 961 bytes} [Sun Oct 16 00:55:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:55:17.289495 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1cf2f3-7a31-4ed4-9ae2-9c9eddb8d9ab None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:17.290078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1cf2f3-7a31-4ed4-9ae2-9c9eddb8d9ab None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:17.290078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1cf2f3-7a31-4ed4-9ae2-9c9eddb8d9ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:17.314166 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4e1cf2f3-7a31-4ed4-9ae2-9c9eddb8d9ab None None] MFA Rules not processed for user `87785ccd37a642e08a0ad34a4fb1c754`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:55:17.315712 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4e1cf2f3-7a31-4ed4-9ae2-9c9eddb8d9ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:55:17.325267 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 587/1197] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Oct 16 00:55:17 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:55:17.334559 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f87960c-ce54-4d9f-807a-9efeb1ae0c9c None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:55:17.348777 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f87960c-ce54-4d9f-807a-9efeb1ae0c9c None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:55:17.354399 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c397896-952b-46ba-8fd5-cb1d4efca9a4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87785ccd37a642e08a0ad34a4fb1c754', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:55:17.355055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c397896-952b-46ba-8fd5-cb1d4efca9a4 service placement] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:17.355335 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c397896-952b-46ba-8fd5-cb1d4efca9a4 service placement] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:17.355608 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c397896-952b-46ba-8fd5-cb1d4efca9a4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:17.366442 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c397896-952b-46ba-8fd5-cb1d4efca9a4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:55:17.368105 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c397896-952b-46ba-8fd5-cb1d4efca9a4 service placement] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:55:17.382220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 611/1198] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:55:17 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:55:24.913959 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886b3661-ca06-4c7f-9240-09156eaccf8c None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:24.914538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886b3661-ca06-4c7f-9240-09156eaccf8c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:24.914538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886b3661-ca06-4c7f-9240-09156eaccf8c None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:24.915908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 588/1199] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 16 00:55:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:55:24.922442 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8926ed6-9e44-43a5-bd41-5333a22c880d None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:24.922953 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8926ed6-9e44-43a5-bd41-5333a22c880d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:24.923058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8926ed6-9e44-43a5-bd41-5333a22c880d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:24.943530 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-d8926ed6-9e44-43a5-bd41-5333a22c880d None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:55:24.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d8926ed6-9e44-43a5-bd41-5333a22c880d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:55:24.953021 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 612/1200] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 16 00:55:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:55:25.108877 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886b3661-ca06-4c7f-9240-09156eaccf8c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:55:25.124008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886b3661-ca06-4c7f-9240-09156eaccf8c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:55:25.130153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce5bff9-ded9-461a-8f50-d1c318a419d5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:55:25.130758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce5bff9-ded9-461a-8f50-d1c318a419d5 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:25.131069 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce5bff9-ded9-461a-8f50-d1c318a419d5 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:25.131348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce5bff9-ded9-461a-8f50-d1c318a419d5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:25.139753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce5bff9-ded9-461a-8f50-d1c318a419d5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:55:25.141358 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce5bff9-ded9-461a-8f50-d1c318a419d5 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:55:25.151592 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 589/1201] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:55:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:55:26.243534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9641faf5-c24e-4e2b-9936-716c954e066e None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:26.244070 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9641faf5-c24e-4e2b-9936-716c954e066e None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:26.244070 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9641faf5-c24e-4e2b-9936-716c954e066e None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:26.245346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 613/1202] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 16 00:55:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:55:26.251545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651af758-811c-4787-9c82-a2de68e98feb None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:26.251970 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651af758-811c-4787-9c82-a2de68e98feb None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:26.252115 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651af758-811c-4787-9c82-a2de68e98feb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:26.273382 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-651af758-811c-4787-9c82-a2de68e98feb None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:55:26.274608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-651af758-811c-4787-9c82-a2de68e98feb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:55:26.282913 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 590/1203] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 16 00:55:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:55:26.438540 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9641faf5-c24e-4e2b-9936-716c954e066e None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:55:26.453093 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9641faf5-c24e-4e2b-9936-716c954e066e None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:55:26.458473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84b8ced-33ab-4f77-ad09-d5176966a972 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:55:26.459105 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84b8ced-33ab-4f77-ad09-d5176966a972 service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:26.459388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84b8ced-33ab-4f77-ad09-d5176966a972 service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:26.459657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84b8ced-33ab-4f77-ad09-d5176966a972 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:26.468135 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84b8ced-33ab-4f77-ad09-d5176966a972 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:55:26.469873 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84b8ced-33ab-4f77-ad09-d5176966a972 service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:55:26.480301 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 614/1204] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:55:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:55:30.582377 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b6f5de-6644-4fa4-b504-72c5c7cb4074 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:30.583023 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b6f5de-6644-4fa4-b504-72c5c7cb4074 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:30.583023 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b6f5de-6644-4fa4-b504-72c5c7cb4074 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:30.584316 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 591/1205] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 16 00:55:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:55:30.589703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3521b9-666a-4608-adee-e077718fda07 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:30.590125 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3521b9-666a-4608-adee-e077718fda07 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:30.590266 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3521b9-666a-4608-adee-e077718fda07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:30.609959 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-5f3521b9-666a-4608-adee-e077718fda07 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:55:30.611203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-5f3521b9-666a-4608-adee-e077718fda07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:55:30.620329 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 615/1206] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 16 00:55:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:55:30.918668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b6f5de-6644-4fa4-b504-72c5c7cb4074 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:55:30.920545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-18b6f5de-6644-4fa4-b504-72c5c7cb4074 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:55:31.001852 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b6f5de-6644-4fa4-b504-72c5c7cb4074 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:55:31.005530 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20386db6-617e-4855-b989-341531c0dae8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:55:31.006140 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20386db6-617e-4855-b989-341531c0dae8 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:31.006420 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20386db6-617e-4855-b989-341531c0dae8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:31.006692 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20386db6-617e-4855-b989-341531c0dae8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:31.016095 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20386db6-617e-4855-b989-341531c0dae8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:55:31.017538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20386db6-617e-4855-b989-341531c0dae8 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:55:31.027072 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 592/1207] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:55:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:55:33.247326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad3a12-1e97-4a3c-9486-b43f66f686cf None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:33.247904 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad3a12-1e97-4a3c-9486-b43f66f686cf None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:33.247904 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad3a12-1e97-4a3c-9486-b43f66f686cf None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:33.249210 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 616/1208] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 16 00:55:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:55:33.255394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80c2ce6-8fe4-4b13-b23e-d0b6051058ec None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:33.255935 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80c2ce6-8fe4-4b13-b23e-d0b6051058ec None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:33.255935 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80c2ce6-8fe4-4b13-b23e-d0b6051058ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:33.276298 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e80c2ce6-8fe4-4b13-b23e-d0b6051058ec None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:55:33.277560 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e80c2ce6-8fe4-4b13-b23e-d0b6051058ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:55:33.285858 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 593/1209] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 16 00:55:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:55:33.573692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ad3a12-1e97-4a3c-9486-b43f66f686cf None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:55:33.575503 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-f0ad3a12-1e97-4a3c-9486-b43f66f686cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:55:33.584076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=75 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:55:33.590445 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ad3a12-1e97-4a3c-9486-b43f66f686cf None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:55:33.594047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819fae4b-297d-4bd6-85fc-6568bd78b85a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:55:33.594652 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819fae4b-297d-4bd6-85fc-6568bd78b85a service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:33.594968 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819fae4b-297d-4bd6-85fc-6568bd78b85a service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:33.595246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819fae4b-297d-4bd6-85fc-6568bd78b85a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:33.604896 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819fae4b-297d-4bd6-85fc-6568bd78b85a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:55:33.606324 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819fae4b-297d-4bd6-85fc-6568bd78b85a service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:55:33.615886 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 617/1210] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:55:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:55:37.415251 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6f8a3b-e55b-40f7-afe2-a127ea1d3388 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:37.415815 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6f8a3b-e55b-40f7-afe2-a127ea1d3388 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:37.415815 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6f8a3b-e55b-40f7-afe2-a127ea1d3388 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:37.417162 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 594/1211] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 16 00:55:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:55:37.422570 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a81edd1-b02a-48b6-b007-fb88d4b5d749 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:37.423043 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a81edd1-b02a-48b6-b007-fb88d4b5d749 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:37.423153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a81edd1-b02a-48b6-b007-fb88d4b5d749 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:37.443358 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-1a81edd1-b02a-48b6-b007-fb88d4b5d749 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:55:37.444574 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-1a81edd1-b02a-48b6-b007-fb88d4b5d749 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:55:37.452719 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 618/1212] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 16 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:55:37.760543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6f8a3b-e55b-40f7-afe2-a127ea1d3388 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:55:37.762235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2a6f8a3b-e55b-40f7-afe2-a127ea1d3388 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:55:37.826610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:55:37.841980 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6f8a3b-e55b-40f7-afe2-a127ea1d3388 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:55:37.845613 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186f980f-e3d5-47bb-a4db-9ad6656b9a20 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:55:37.846219 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186f980f-e3d5-47bb-a4db-9ad6656b9a20 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:37.846500 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186f980f-e3d5-47bb-a4db-9ad6656b9a20 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:37.846779 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186f980f-e3d5-47bb-a4db-9ad6656b9a20 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:37.855897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186f980f-e3d5-47bb-a4db-9ad6656b9a20 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:55:37.857334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186f980f-e3d5-47bb-a4db-9ad6656b9a20 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:55:37.866732 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 595/1213] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:55:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:55:40.035628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72afed9a-a29b-46a1-8e00-d069e68ab6e6 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:40.036356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72afed9a-a29b-46a1-8e00-d069e68ab6e6 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:40.036356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72afed9a-a29b-46a1-8e00-d069e68ab6e6 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:40.037549 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 619/1214] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 16 00:55:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:55:40.042783 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e9d1fb-9d23-489d-aed8-8721d6b87a9e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:40.043242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e9d1fb-9d23-489d-aed8-8721d6b87a9e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:40.043358 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e9d1fb-9d23-489d-aed8-8721d6b87a9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:40.062746 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a4e9d1fb-9d23-489d-aed8-8721d6b87a9e None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:55:40.063980 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a4e9d1fb-9d23-489d-aed8-8721d6b87a9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:55:40.072390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 596/1215] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 16 00:55:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:55:40.381180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72afed9a-a29b-46a1-8e00-d069e68ab6e6 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:55:40.382935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-72afed9a-a29b-46a1-8e00-d069e68ab6e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:55:40.396450 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72afed9a-a29b-46a1-8e00-d069e68ab6e6 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:55:40.400188 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4526541-efed-4124-8c4c-895b8a2829df service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:55:40.400800 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4526541-efed-4124-8c4c-895b8a2829df service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:55:40.401081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4526541-efed-4124-8c4c-895b8a2829df service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:55:40.401353 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4526541-efed-4124-8c4c-895b8a2829df service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:55:40.411307 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4526541-efed-4124-8c4c-895b8a2829df service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:55:40.412795 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4526541-efed-4124-8c4c-895b8a2829df service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:55:40.422773 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 620/1216] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:55:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:55:50.057823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=11 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:55:50.418209 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=14 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:09.526502 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5b8e65-5e64-4db1-9b42-8dd720fc96d4 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:09.527577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5b8e65-5e64-4db1-9b42-8dd720fc96d4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:09.527577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5b8e65-5e64-4db1-9b42-8dd720fc96d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:09.556113 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5e5b8e65-5e64-4db1-9b42-8dd720fc96d4 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:09.557556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5e5b8e65-5e64-4db1-9b42-8dd720fc96d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:09.570179 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 597/1217] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:09.598268 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9432ecb3-ad46-46f2-9298-634d6744e39b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:09.599852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9432ecb3-ad46-46f2-9298-634d6744e39b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:09.600658 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9432ecb3-ad46-46f2-9298-634d6744e39b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:09.638848 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-9432ecb3-ad46-46f2-9298-634d6744e39b None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:09.641262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-9432ecb3-ad46-46f2-9298-634d6744e39b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:09.656448 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 621/1218] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:09.663714 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5b8e65-5e64-4db1-9b42-8dd720fc96d4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:09.681750 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5b8e65-5e64-4db1-9b42-8dd720fc96d4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:09.690049 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56411144-70c0-4b2d-b180-671b0010b7bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:09.690601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56411144-70c0-4b2d-b180-671b0010b7bf admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:09.691017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56411144-70c0-4b2d-b180-671b0010b7bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:09.691294 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56411144-70c0-4b2d-b180-671b0010b7bf admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:09.692128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56411144-70c0-4b2d-b180-671b0010b7bf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:57:09.692457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56411144-70c0-4b2d-b180-671b0010b7bf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:09.694199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56411144-70c0-4b2d-b180-671b0010b7bf admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:09.722544 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 598/1219] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:57:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:57:09.732755 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9432ecb3-ad46-46f2-9298-634d6744e39b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:09.734799 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3365ee3-2b77-4697-abda-fe47c090e39c None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:09.735462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3365ee3-2b77-4697-abda-fe47c090e39c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:09.735997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3365ee3-2b77-4697-abda-fe47c090e39c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:09.750482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9432ecb3-ad46-46f2-9298-634d6744e39b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:09.760417 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606ab53f-f767-4d93-ad33-d572d020a2ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:09.761065 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606ab53f-f767-4d93-ad33-d572d020a2ab admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:09.761349 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606ab53f-f767-4d93-ad33-d572d020a2ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:09.761635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606ab53f-f767-4d93-ad33-d572d020a2ab admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:09.762278 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606ab53f-f767-4d93-ad33-d572d020a2ab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:09.765039 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606ab53f-f767-4d93-ad33-d572d020a2ab admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:09.769446 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f3365ee3-2b77-4697-abda-fe47c090e39c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:09.772378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f3365ee3-2b77-4697-abda-fe47c090e39c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:09.814935 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 599/1220] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:57:09.835104 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 622/1221] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:09 2022] POST /identity/v3/projects => generated 379 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:09.840235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6458c6ab-090a-41b4-a20c-8f79c78ca617 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:09.840727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6458c6ab-090a-41b4-a20c-8f79c78ca617 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:09.841218 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6458c6ab-090a-41b4-a20c-8f79c78ca617 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:09.851709 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606ab53f-f767-4d93-ad33-d572d020a2ab admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:09.929953 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-6458c6ab-090a-41b4-a20c-8f79c78ca617 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:09.936633 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6458c6ab-090a-41b4-a20c-8f79c78ca617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:10.013708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606ab53f-f767-4d93-ad33-d572d020a2ab admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:10.023136 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48eab84-6111-4d31-aab6-83510e48c8c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:10.024239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48eab84-6111-4d31-aab6-83510e48c8c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.024751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48eab84-6111-4d31-aab6-83510e48c8c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.025244 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48eab84-6111-4d31-aab6-83510e48c8c8 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.026229 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48eab84-6111-4d31-aab6-83510e48c8c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:10.027576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: WARNING py.warnings [None req-e48eab84-6111-4d31-aab6-83510e48c8c8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 16 00:57:10.027576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: warnings.warn(msg) Oct 16 00:57:10.027576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: Oct 16 00:57:10.033916 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48eab84-6111-4d31-aab6-83510e48c8c8 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:10.083079 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 623/1222] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:09 2022] POST /identity/v3/users => generated 290 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:57:10.084846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067659e9-3db6-4784-825a-5dfe64f195df None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.085416 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067659e9-3db6-4784-825a-5dfe64f195df None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.085915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067659e9-3db6-4784-825a-5dfe64f195df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.114274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 600/1223] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:10.116062 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba786ecd-8b4b-431c-92ab-072ccf001b04 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.116929 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba786ecd-8b4b-431c-92ab-072ccf001b04 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.118982 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba786ecd-8b4b-431c-92ab-072ccf001b04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.126954 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-067659e9-3db6-4784-825a-5dfe64f195df None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:10.129255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-067659e9-3db6-4784-825a-5dfe64f195df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:10.149254 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 624/1224] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:10.156485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401ee924-c9ca-45d9-bbbc-cc6740daafa9 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.157261 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401ee924-c9ca-45d9-bbbc-cc6740daafa9 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.157996 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401ee924-c9ca-45d9-bbbc-cc6740daafa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.163132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ba786ecd-8b4b-431c-92ab-072ccf001b04 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:10.165144 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ba786ecd-8b4b-431c-92ab-072ccf001b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:10.184538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 601/1225] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:10.185889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba786ecd-8b4b-431c-92ab-072ccf001b04 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:10.200058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-401ee924-c9ca-45d9-bbbc-cc6740daafa9 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:10.204477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-401ee924-c9ca-45d9-bbbc-cc6740daafa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:10.208403 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba786ecd-8b4b-431c-92ab-072ccf001b04 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:10.217023 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0718197b-77e6-46c9-b0c5-7e564423aac3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:10.218605 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0718197b-77e6-46c9-b0c5-7e564423aac3 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.219366 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0718197b-77e6-46c9-b0c5-7e564423aac3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.223464 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0718197b-77e6-46c9-b0c5-7e564423aac3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.225009 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0718197b-77e6-46c9-b0c5-7e564423aac3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:10.234027 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0718197b-77e6-46c9-b0c5-7e564423aac3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:10.237909 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 625/1226] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:10.239869 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401ee924-c9ca-45d9-bbbc-cc6740daafa9 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:10.257654 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 602/1227] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:10 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:10.259458 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9588a16f-3cfe-46fb-968c-7fb2c9e90495 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.259925 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9588a16f-3cfe-46fb-968c-7fb2c9e90495 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.260399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9588a16f-3cfe-46fb-968c-7fb2c9e90495 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.271663 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401ee924-c9ca-45d9-bbbc-cc6740daafa9 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:10.286292 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8e5ad0-1659-4fca-8d97-7d8cd464e223 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:10.287570 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8e5ad0-1659-4fca-8d97-7d8cd464e223 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.288162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8e5ad0-1659-4fca-8d97-7d8cd464e223 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.288727 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8e5ad0-1659-4fca-8d97-7d8cd464e223 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.290021 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8e5ad0-1659-4fca-8d97-7d8cd464e223 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:57:10.290782 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8e5ad0-1659-4fca-8d97-7d8cd464e223 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:10.292694 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: WARNING py.warnings [None req-2d8e5ad0-1659-4fca-8d97-7d8cd464e223 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 16 00:57:10.292694 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: warnings.warn(msg) Oct 16 00:57:10.292694 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: Oct 16 00:57:10.296482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8e5ad0-1659-4fca-8d97-7d8cd464e223 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:10.298148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-9588a16f-3cfe-46fb-968c-7fb2c9e90495 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:10.300447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9588a16f-3cfe-46fb-968c-7fb2c9e90495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:10.315504 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 603/1228] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:10.317274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3ff150-3f97-40da-bebf-6493b60ad2d9 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.317601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3ff150-3f97-40da-bebf-6493b60ad2d9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.317919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3ff150-3f97-40da-bebf-6493b60ad2d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.322941 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 626/1229] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:57:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:57:10.325467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d51ac60-4e37-42c6-8f37-3cd71be99e81 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.325849 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d51ac60-4e37-42c6-8f37-3cd71be99e81 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.326192 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d51ac60-4e37-42c6-8f37-3cd71be99e81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.345244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-db3ff150-3f97-40da-bebf-6493b60ad2d9 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:10.346828 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-db3ff150-3f97-40da-bebf-6493b60ad2d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:10.350456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-0d51ac60-4e37-42c6-8f37-3cd71be99e81 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:10.351723 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-0d51ac60-4e37-42c6-8f37-3cd71be99e81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:10.360289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 604/1230] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:10.360289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a50006c-31fd-4841-90ee-2b572f14a691 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.360289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a50006c-31fd-4841-90ee-2b572f14a691 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.360289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a50006c-31fd-4841-90ee-2b572f14a691 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.364547 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 627/1231] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:10.365478 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d51ac60-4e37-42c6-8f37-3cd71be99e81 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:10.377147 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d51ac60-4e37-42c6-8f37-3cd71be99e81 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:10.382186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50561638-2313-42d4-ae31-4f453b2ef377 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:10.382918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50561638-2313-42d4-ae31-4f453b2ef377 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.383206 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50561638-2313-42d4-ae31-4f453b2ef377 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.383478 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50561638-2313-42d4-ae31-4f453b2ef377 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.384132 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50561638-2313-42d4-ae31-4f453b2ef377 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:10.385803 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50561638-2313-42d4-ae31-4f453b2ef377 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:10.401578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 628/1232] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:10 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:10.401873 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3a50006c-31fd-4841-90ee-2b572f14a691 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:10.402049 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21015397-e15e-458a-a23d-fd551ae215a3 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.402049 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21015397-e15e-458a-a23d-fd551ae215a3 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.402049 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21015397-e15e-458a-a23d-fd551ae215a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.402532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3a50006c-31fd-4841-90ee-2b572f14a691 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:10.413160 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 605/1233] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:10.415384 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8406ee-d547-4a0a-965e-cf73c70bc30d None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.415898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8406ee-d547-4a0a-965e-cf73c70bc30d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.416407 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8406ee-d547-4a0a-965e-cf73c70bc30d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.424899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-21015397-e15e-458a-a23d-fd551ae215a3 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:10.426132 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-21015397-e15e-458a-a23d-fd551ae215a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:10.437650 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 629/1234] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:10.439133 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ee0ec7-080f-432c-98b6-40db94d72c42 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.439488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ee0ec7-080f-432c-98b6-40db94d72c42 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.439776 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ee0ec7-080f-432c-98b6-40db94d72c42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.454186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3d8406ee-d547-4a0a-965e-cf73c70bc30d None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:10.456477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3d8406ee-d547-4a0a-965e-cf73c70bc30d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:10.463221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-32ee0ec7-080f-432c-98b6-40db94d72c42 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:10.464511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-32ee0ec7-080f-432c-98b6-40db94d72c42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:10.469126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 606/1235] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:10.469977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8406ee-d547-4a0a-965e-cf73c70bc30d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:10.478145 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 630/1236] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:10.479043 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ee0ec7-080f-432c-98b6-40db94d72c42 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:10.488910 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8406ee-d547-4a0a-965e-cf73c70bc30d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:10.489309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ee0ec7-080f-432c-98b6-40db94d72c42 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:10.493946 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7092a9c-cfd0-42f3-8297-0d871336d71c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:10.494204 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f34505-06e4-4b1b-b2be-d06840a9756a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:10.494564 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7092a9c-cfd0-42f3-8297-0d871336d71c admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.494816 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f34505-06e4-4b1b-b2be-d06840a9756a admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.494990 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7092a9c-cfd0-42f3-8297-0d871336d71c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.495194 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f34505-06e4-4b1b-b2be-d06840a9756a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.495399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7092a9c-cfd0-42f3-8297-0d871336d71c admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.495515 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f34505-06e4-4b1b-b2be-d06840a9756a admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.495949 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7092a9c-cfd0-42f3-8297-0d871336d71c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:10.496148 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f34505-06e4-4b1b-b2be-d06840a9756a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:57:10.496313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f34505-06e4-4b1b-b2be-d06840a9756a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:10.497011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: WARNING py.warnings [None req-e7092a9c-cfd0-42f3-8297-0d871336d71c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 16 00:57:10.497011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: warnings.warn(msg) Oct 16 00:57:10.497011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: Oct 16 00:57:10.498095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f34505-06e4-4b1b-b2be-d06840a9756a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:10.498332 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7092a9c-cfd0-42f3-8297-0d871336d71c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:10.511426 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 631/1237] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:57:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:57:10.512288 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f34505-06e4-4b1b-b2be-d06840a9756a admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:10.523399 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f34505-06e4-4b1b-b2be-d06840a9756a admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:10.529091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6856b52b-0358-4de9-8305-dd512e3a827c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:10.530534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6856b52b-0358-4de9-8305-dd512e3a827c admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.531190 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6856b52b-0358-4de9-8305-dd512e3a827c admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.531897 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6856b52b-0358-4de9-8305-dd512e3a827c admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.532640 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6856b52b-0358-4de9-8305-dd512e3a827c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:57:10.533039 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6856b52b-0358-4de9-8305-dd512e3a827c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:10.535037 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6856b52b-0358-4de9-8305-dd512e3a827c admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:10.546699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 607/1238] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:10 2022] POST /identity/v3/projects => generated 373 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:10.547898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7092a9c-cfd0-42f3-8297-0d871336d71c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:10.552575 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 632/1239] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:57:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:57:10.552575 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6856b52b-0358-4de9-8305-dd512e3a827c admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:10.563259 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6856b52b-0358-4de9-8305-dd512e3a827c admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:10.598740 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:10.640996 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68767c5a-fefd-4866-82ac-698672bf378f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:10.641780 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68767c5a-fefd-4866-82ac-698672bf378f admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.642063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68767c5a-fefd-4866-82ac-698672bf378f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.642336 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68767c5a-fefd-4866-82ac-698672bf378f admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.643069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68767c5a-fefd-4866-82ac-698672bf378f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:57:10.643581 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68767c5a-fefd-4866-82ac-698672bf378f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:10.645220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68767c5a-fefd-4866-82ac-698672bf378f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:10.646796 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7092a9c-cfd0-42f3-8297-0d871336d71c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:10.651890 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b77632-20d2-4053-a552-46c965caad3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:10.652610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b77632-20d2-4053-a552-46c965caad3b admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.652899 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b77632-20d2-4053-a552-46c965caad3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.653181 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b77632-20d2-4053-a552-46c965caad3b admin admin] PATH_INFO: `/v3/projects/c89fd0e945344feab91a37fdb93812e5/users/3aa7409c9f6f46f78164818b6f6927e4/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.658389 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 633/1240] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:57:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:57:10.659791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e126ad28-641f-4c2b-880b-52163f2f68b1 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.660084 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e126ad28-641f-4c2b-880b-52163f2f68b1 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.660357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e126ad28-641f-4c2b-880b-52163f2f68b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.672792 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:10.676954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b77632-20d2-4053-a552-46c965caad3b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c89fd0e945344feab91a37fdb93812e5, user_id=3aa7409c9f6f46f78164818b6f6927e4, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:10.680653 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b77632-20d2-4053-a552-46c965caad3b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:10.681794 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-e126ad28-641f-4c2b-880b-52163f2f68b1 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:10.683078 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-e126ad28-641f-4c2b-880b-52163f2f68b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:10.693834 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 634/1241] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:10.695011 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e126ad28-641f-4c2b-880b-52163f2f68b1 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:10.704881 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 608/1242] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:10 2022] PUT /identity/v3/projects/c89fd0e945344feab91a37fdb93812e5/users/3aa7409c9f6f46f78164818b6f6927e4/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:10.706935 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1151cecf-41ff-4db8-8353-52cb11e41b1e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.707128 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e126ad28-641f-4c2b-880b-52163f2f68b1 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:10.707546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1151cecf-41ff-4db8-8353-52cb11e41b1e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.707966 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1151cecf-41ff-4db8-8353-52cb11e41b1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.735982 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-1151cecf-41ff-4db8-8353-52cb11e41b1e None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:10.737977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1151cecf-41ff-4db8-8353-52cb11e41b1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:10.777986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff22d85-1ecc-4939-b3d3-36c2460b2d16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:10.778960 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff22d85-1ecc-4939-b3d3-36c2460b2d16 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.779424 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff22d85-1ecc-4939-b3d3-36c2460b2d16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.779930 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff22d85-1ecc-4939-b3d3-36c2460b2d16 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.781094 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff22d85-1ecc-4939-b3d3-36c2460b2d16 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:57:10.781607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff22d85-1ecc-4939-b3d3-36c2460b2d16 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:10.784557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff22d85-1ecc-4939-b3d3-36c2460b2d16 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:10.802278 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 635/1243] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:57:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:57:10.803155 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff22d85-1ecc-4939-b3d3-36c2460b2d16 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:10.817814 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff22d85-1ecc-4939-b3d3-36c2460b2d16 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:10.828360 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc5e1ab-25d3-41df-a929-b5ba53454eb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:10.829147 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc5e1ab-25d3-41df-a929-b5ba53454eb4 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.829556 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc5e1ab-25d3-41df-a929-b5ba53454eb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.829927 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc5e1ab-25d3-41df-a929-b5ba53454eb4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.830672 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc5e1ab-25d3-41df-a929-b5ba53454eb4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:10.832781 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc5e1ab-25d3-41df-a929-b5ba53454eb4 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:10.833034 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 609/1244] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:10.834279 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1151cecf-41ff-4db8-8353-52cb11e41b1e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:10.850233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1151cecf-41ff-4db8-8353-52cb11e41b1e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:10.859572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3e358b-2e65-4cab-b93d-0eeb7c28007c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:10.860505 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3e358b-2e65-4cab-b93d-0eeb7c28007c admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.860991 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3e358b-2e65-4cab-b93d-0eeb7c28007c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.861465 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3e358b-2e65-4cab-b93d-0eeb7c28007c admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.862889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3e358b-2e65-4cab-b93d-0eeb7c28007c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:10.864558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 636/1245] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:10 2022] POST /identity/v3/projects => generated 405 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:10.865392 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc5e1ab-25d3-41df-a929-b5ba53454eb4 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:10.866178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3e358b-2e65-4cab-b93d-0eeb7c28007c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:10.906304 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 610/1246] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:10 2022] POST /identity/v3/users => generated 287 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:10.907692 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3e358b-2e65-4cab-b93d-0eeb7c28007c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:10.924361 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3e358b-2e65-4cab-b93d-0eeb7c28007c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:10.950498 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc5e1ab-25d3-41df-a929-b5ba53454eb4 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:10.957988 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2704e942-9a8a-43de-b96a-75e284628e63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:10.958651 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2704e942-9a8a-43de-b96a-75e284628e63 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:10.958977 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2704e942-9a8a-43de-b96a-75e284628e63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:10.959264 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2704e942-9a8a-43de-b96a-75e284628e63 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:10.959928 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2704e942-9a8a-43de-b96a-75e284628e63 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:10.961933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2704e942-9a8a-43de-b96a-75e284628e63 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.001126 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 637/1247] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:10 2022] POST /identity/v3/projects => generated 389 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:11.001126 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2704e942-9a8a-43de-b96a-75e284628e63 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.003496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f984a1-2147-4739-b06b-1a8f153b366e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.004537 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f984a1-2147-4739-b06b-1a8f153b366e admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.004958 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f984a1-2147-4739-b06b-1a8f153b366e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.005470 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f984a1-2147-4739-b06b-1a8f153b366e admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.006476 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f984a1-2147-4739-b06b-1a8f153b366e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.009916 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f984a1-2147-4739-b06b-1a8f153b366e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.012811 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:11.052675 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 611/1248] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:10 2022] POST /identity/v3/projects => generated 393 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:11.054008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1933a906-bf8f-47d6-83b8-e4c856cc7ef7 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.054301 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1933a906-bf8f-47d6-83b8-e4c856cc7ef7 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.054578 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1933a906-bf8f-47d6-83b8-e4c856cc7ef7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.084009 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-1933a906-bf8f-47d6-83b8-e4c856cc7ef7 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:11.085698 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1933a906-bf8f-47d6-83b8-e4c856cc7ef7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:11.086817 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2704e942-9a8a-43de-b96a-75e284628e63 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.152998 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff7e8e8-be4a-4019-8072-884a721daf86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.153626 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff7e8e8-be4a-4019-8072-884a721daf86 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.153905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff7e8e8-be4a-4019-8072-884a721daf86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.154177 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff7e8e8-be4a-4019-8072-884a721daf86 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.154611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:11.155854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff7e8e8-be4a-4019-8072-884a721daf86 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:57:11.155854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff7e8e8-be4a-4019-8072-884a721daf86 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.157413 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff7e8e8-be4a-4019-8072-884a721daf86 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.170104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 612/1249] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:11.170532 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 638/1250] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:57:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:57:11.178118 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff7e8e8-be4a-4019-8072-884a721daf86 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.178780 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1933a906-bf8f-47d6-83b8-e4c856cc7ef7 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.190072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff7e8e8-be4a-4019-8072-884a721daf86 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.192321 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1933a906-bf8f-47d6-83b8-e4c856cc7ef7 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.195435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89680844-ca2e-432b-b08c-adf0aa02c604 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.196069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89680844-ca2e-432b-b08c-adf0aa02c604 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.196346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89680844-ca2e-432b-b08c-adf0aa02c604 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.196625 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89680844-ca2e-432b-b08c-adf0aa02c604 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.197313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89680844-ca2e-432b-b08c-adf0aa02c604 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:57:11.197624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89680844-ca2e-432b-b08c-adf0aa02c604 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.199337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89680844-ca2e-432b-b08c-adf0aa02c604 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.200602 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba2712b-7a9e-405c-be8f-901ee2e3e4e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.202431 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba2712b-7a9e-405c-be8f-901ee2e3e4e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.202956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba2712b-7a9e-405c-be8f-901ee2e3e4e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.203451 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba2712b-7a9e-405c-be8f-901ee2e3e4e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.204669 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba2712b-7a9e-405c-be8f-901ee2e3e4e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.208662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba2712b-7a9e-405c-be8f-901ee2e3e4e7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.222204 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 639/1251] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:57:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:57:11.222204 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89680844-ca2e-432b-b08c-adf0aa02c604 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.228126 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89680844-ca2e-432b-b08c-adf0aa02c604 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.233933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6100a131-46a1-43c0-bddb-c3f359f10e00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.234616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6100a131-46a1-43c0-bddb-c3f359f10e00 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.234958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6100a131-46a1-43c0-bddb-c3f359f10e00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.235429 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6100a131-46a1-43c0-bddb-c3f359f10e00 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.235972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6100a131-46a1-43c0-bddb-c3f359f10e00 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.238070 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6100a131-46a1-43c0-bddb-c3f359f10e00 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.249663 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 613/1252] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:11 2022] POST /identity/v3/projects => generated 383 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:11.250620 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba2712b-7a9e-405c-be8f-901ee2e3e4e7 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.263504 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 640/1253] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:11 2022] POST /identity/v3/users => generated 303 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:11.264916 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6100a131-46a1-43c0-bddb-c3f359f10e00 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.275635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6100a131-46a1-43c0-bddb-c3f359f10e00 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.333610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba2712b-7a9e-405c-be8f-901ee2e3e4e7 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.339843 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626fdf97-6503-4c72-8a0c-bcf7b4dace8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.340646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626fdf97-6503-4c72-8a0c-bcf7b4dace8b admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.341067 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626fdf97-6503-4c72-8a0c-bcf7b4dace8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.341487 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626fdf97-6503-4c72-8a0c-bcf7b4dace8b admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.343319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626fdf97-6503-4c72-8a0c-bcf7b4dace8b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.346038 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f688a27-9191-43b8-970c-9d108c67106d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.346326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626fdf97-6503-4c72-8a0c-bcf7b4dace8b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.346672 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f688a27-9191-43b8-970c-9d108c67106d admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.346987 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f688a27-9191-43b8-970c-9d108c67106d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.347262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f688a27-9191-43b8-970c-9d108c67106d admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.347905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f688a27-9191-43b8-970c-9d108c67106d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.349890 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f688a27-9191-43b8-970c-9d108c67106d admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.362405 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 614/1254] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:11 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:11.362405 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626fdf97-6503-4c72-8a0c-bcf7b4dace8b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.376608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626fdf97-6503-4c72-8a0c-bcf7b4dace8b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.383048 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 641/1255] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:11 2022] POST /identity/v3/users => generated 295 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:11.383633 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e507e794-dbb0-462b-94c4-850cb7c1432c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.384822 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e507e794-dbb0-462b-94c4-850cb7c1432c admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.385350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e507e794-dbb0-462b-94c4-850cb7c1432c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.386160 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e507e794-dbb0-462b-94c4-850cb7c1432c admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.387856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f688a27-9191-43b8-970c-9d108c67106d admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.388142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e507e794-dbb0-462b-94c4-850cb7c1432c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.392021 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e507e794-dbb0-462b-94c4-850cb7c1432c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.401314 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f688a27-9191-43b8-970c-9d108c67106d admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.408868 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ae62d4-f3f3-4dc8-9dd2-d61de02050c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.409862 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ae62d4-f3f3-4dc8-9dd2-d61de02050c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.410305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ae62d4-f3f3-4dc8-9dd2-d61de02050c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.410695 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ae62d4-f3f3-4dc8-9dd2-d61de02050c5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.412038 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ae62d4-f3f3-4dc8-9dd2-d61de02050c5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.415798 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ae62d4-f3f3-4dc8-9dd2-d61de02050c5 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.426916 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 615/1256] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:11 2022] POST /identity/v3/users => generated 297 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:11.426916 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e507e794-dbb0-462b-94c4-850cb7c1432c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.435170 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:11.463359 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 642/1257] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:11 2022] POST /identity/v3/projects => generated 371 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:11.464859 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ae62d4-f3f3-4dc8-9dd2-d61de02050c5 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.499022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e507e794-dbb0-462b-94c4-850cb7c1432c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.560050 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38901bb-ac8b-452a-9377-f8aa70f666e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.562000 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38901bb-ac8b-452a-9377-f8aa70f666e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.562390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38901bb-ac8b-452a-9377-f8aa70f666e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.562774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38901bb-ac8b-452a-9377-f8aa70f666e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.573599 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38901bb-ac8b-452a-9377-f8aa70f666e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.573890 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ae62d4-f3f3-4dc8-9dd2-d61de02050c5 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.575405 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38901bb-ac8b-452a-9377-f8aa70f666e8 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.580159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9a833e-c77c-4910-9a06-6e80bae9d313 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.580948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9a833e-c77c-4910-9a06-6e80bae9d313 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.581235 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9a833e-c77c-4910-9a06-6e80bae9d313 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.581514 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9a833e-c77c-4910-9a06-6e80bae9d313 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.582172 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9a833e-c77c-4910-9a06-6e80bae9d313 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.584272 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9a833e-c77c-4910-9a06-6e80bae9d313 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.587513 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 616/1258] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:11.588371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38901bb-ac8b-452a-9377-f8aa70f666e8 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.599710 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38901bb-ac8b-452a-9377-f8aa70f666e8 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.604790 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea5f0a9-caa4-411d-9fe5-70c9004dfae1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.605827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea5f0a9-caa4-411d-9fe5-70c9004dfae1 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.606139 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea5f0a9-caa4-411d-9fe5-70c9004dfae1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.606438 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea5f0a9-caa4-411d-9fe5-70c9004dfae1 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.607388 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea5f0a9-caa4-411d-9fe5-70c9004dfae1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.610729 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea5f0a9-caa4-411d-9fe5-70c9004dfae1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.620903 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 643/1259] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:11 2022] POST /identity/v3/projects => generated 365 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:11.621814 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9a833e-c77c-4910-9a06-6e80bae9d313 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.634105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:11.645936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 617/1260] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:11 2022] POST /identity/v3/users => generated 292 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:11.646945 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea5f0a9-caa4-411d-9fe5-70c9004dfae1 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.658318 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea5f0a9-caa4-411d-9fe5-70c9004dfae1 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.736776 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c255bc90-d563-4539-a0f9-9e463770c08c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.737536 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c255bc90-d563-4539-a0f9-9e463770c08c admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.737701 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c255bc90-d563-4539-a0f9-9e463770c08c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.737971 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c255bc90-d563-4539-a0f9-9e463770c08c admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.738739 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c255bc90-d563-4539-a0f9-9e463770c08c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.740477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c255bc90-d563-4539-a0f9-9e463770c08c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.755021 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 618/1261] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:11 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:11.755993 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c255bc90-d563-4539-a0f9-9e463770c08c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.766736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9a833e-c77c-4910-9a06-6e80bae9d313 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.770460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c255bc90-d563-4539-a0f9-9e463770c08c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.772875 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f64494c-431d-4b35-b23a-1702e1cd79b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.773516 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f64494c-431d-4b35-b23a-1702e1cd79b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.773798 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f64494c-431d-4b35-b23a-1702e1cd79b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.774069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f64494c-431d-4b35-b23a-1702e1cd79b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.774725 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f64494c-431d-4b35-b23a-1702e1cd79b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.775643 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4175b0-a290-474f-9908-b0df3e35a781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.776373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4175b0-a290-474f-9908-b0df3e35a781 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.776504 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f64494c-431d-4b35-b23a-1702e1cd79b4 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.776870 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4175b0-a290-474f-9908-b0df3e35a781 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.777242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4175b0-a290-474f-9908-b0df3e35a781 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.778037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4175b0-a290-474f-9908-b0df3e35a781 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.780296 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4175b0-a290-474f-9908-b0df3e35a781 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.789405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 644/1262] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:11 2022] GET /identity/v3/roles => generated 1420 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:11.792191 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f64494c-431d-4b35-b23a-1702e1cd79b4 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.803947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 619/1263] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:11 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:11.805211 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4175b0-a290-474f-9908-b0df3e35a781 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.806766 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f64494c-431d-4b35-b23a-1702e1cd79b4 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.813374 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5be31ac-1568-40e1-9e92-5eff14d600d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.815212 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5be31ac-1568-40e1-9e92-5eff14d600d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.816701 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5be31ac-1568-40e1-9e92-5eff14d600d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.817223 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5be31ac-1568-40e1-9e92-5eff14d600d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.818137 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5be31ac-1568-40e1-9e92-5eff14d600d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.820570 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5be31ac-1568-40e1-9e92-5eff14d600d3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.823790 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4175b0-a290-474f-9908-b0df3e35a781 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.832679 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9532bf42-7974-4497-a6fc-0cce9efaa74f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.833694 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9532bf42-7974-4497-a6fc-0cce9efaa74f admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.834208 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9532bf42-7974-4497-a6fc-0cce9efaa74f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.834756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9532bf42-7974-4497-a6fc-0cce9efaa74f admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.835951 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9532bf42-7974-4497-a6fc-0cce9efaa74f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.837949 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 645/1264] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:11 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:11.839358 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9532bf42-7974-4497-a6fc-0cce9efaa74f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.840599 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5be31ac-1568-40e1-9e92-5eff14d600d3 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.854249 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5be31ac-1568-40e1-9e92-5eff14d600d3 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.862293 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3670c010-658c-4943-ba90-b25aecb2093f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.863078 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3670c010-658c-4943-ba90-b25aecb2093f admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.863966 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3670c010-658c-4943-ba90-b25aecb2093f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.863966 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3670c010-658c-4943-ba90-b25aecb2093f admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.865232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3670c010-658c-4943-ba90-b25aecb2093f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.878285 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 620/1265] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:11 2022] POST /identity/v3/users => generated 286 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:11.878285 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9532bf42-7974-4497-a6fc-0cce9efaa74f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.881613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3670c010-658c-4943-ba90-b25aecb2093f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.889357 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9532bf42-7974-4497-a6fc-0cce9efaa74f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.896371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daab3083-9236-455e-ab4a-31baf7a088cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.897320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daab3083-9236-455e-ab4a-31baf7a088cd admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.897740 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daab3083-9236-455e-ab4a-31baf7a088cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.898126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daab3083-9236-455e-ab4a-31baf7a088cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.899106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daab3083-9236-455e-ab4a-31baf7a088cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.900866 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daab3083-9236-455e-ab4a-31baf7a088cd admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.914912 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 646/1266] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:11 2022] POST /identity/v3/users => generated 283 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:11.914912 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3670c010-658c-4943-ba90-b25aecb2093f admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 621/1267] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:11 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:11.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daab3083-9236-455e-ab4a-31baf7a088cd admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.932646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daab3083-9236-455e-ab4a-31baf7a088cd admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.934392 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3670c010-658c-4943-ba90-b25aecb2093f admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.939327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f3ec2a-c00e-4f8a-bb0b-c2c235263c78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.940242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f3ec2a-c00e-4f8a-bb0b-c2c235263c78 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.940242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f3ec2a-c00e-4f8a-bb0b-c2c235263c78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.940659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f3ec2a-c00e-4f8a-bb0b-c2c235263c78 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.941349 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f3ec2a-c00e-4f8a-bb0b-c2c235263c78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.943060 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f3ec2a-c00e-4f8a-bb0b-c2c235263c78 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.943383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cce896-962d-4656-bce3-44f60df26db3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:11.944532 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cce896-962d-4656-bce3-44f60df26db3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:11.945027 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cce896-962d-4656-bce3-44f60df26db3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:11.946326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cce896-962d-4656-bce3-44f60df26db3 admin admin] PATH_INFO: `/v3/projects/0d378a01be2044faa872ca43f816dbd9/users/ca7148520e6b4c3e8f5400a2d79ebc87/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:11.969057 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 622/1268] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:11 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:11.971031 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f3ec2a-c00e-4f8a-bb0b-c2c235263c78 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:11.972164 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cce896-962d-4656-bce3-44f60df26db3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d378a01be2044faa872ca43f816dbd9, user_id=ca7148520e6b4c3e8f5400a2d79ebc87, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:11.975099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cce896-962d-4656-bce3-44f60df26db3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:11.990098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f3ec2a-c00e-4f8a-bb0b-c2c235263c78 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:11.997122 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 647/1269] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:11 2022] PUT /identity/v3/projects/0d378a01be2044faa872ca43f816dbd9/users/ca7148520e6b4c3e8f5400a2d79ebc87/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:11.998720 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cce896-962d-4656-bce3-44f60df26db3 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:12.039036 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:12.101796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cce896-962d-4656-bce3-44f60df26db3 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:12.106163 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec080fd-a5c6-4914-9c56-9586470d1a38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:12.106940 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec080fd-a5c6-4914-9c56-9586470d1a38 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:12.107249 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9a4d8e-0b30-47df-a912-c44e8884ae52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:12.107414 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec080fd-a5c6-4914-9c56-9586470d1a38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:12.107718 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec080fd-a5c6-4914-9c56-9586470d1a38 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:12.108641 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9a4d8e-0b30-47df-a912-c44e8884ae52 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:12.109046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9a4d8e-0b30-47df-a912-c44e8884ae52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:12.109243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec080fd-a5c6-4914-9c56-9586470d1a38 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:12.109609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9a4d8e-0b30-47df-a912-c44e8884ae52 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:12.110420 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9a4d8e-0b30-47df-a912-c44e8884ae52 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:12.111244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec080fd-a5c6-4914-9c56-9586470d1a38 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:12.112345 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9a4d8e-0b30-47df-a912-c44e8884ae52 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:12.127705 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 623/1270] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:11 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:12.128802 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec080fd-a5c6-4914-9c56-9586470d1a38 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:12.134264 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 648/1271] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:11 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:12.135549 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9a4d8e-0b30-47df-a912-c44e8884ae52 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:12.141588 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec080fd-a5c6-4914-9c56-9586470d1a38 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:12.150163 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9b1585-be68-4629-ba32-8cd8090bb44c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:12.150812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9b1585-be68-4629-ba32-8cd8090bb44c admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:12.151284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9b1585-be68-4629-ba32-8cd8090bb44c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:12.151641 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9b1585-be68-4629-ba32-8cd8090bb44c admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:12.152404 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9b1585-be68-4629-ba32-8cd8090bb44c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:12.153138 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9a4d8e-0b30-47df-a912-c44e8884ae52 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:12.155596 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9b1585-be68-4629-ba32-8cd8090bb44c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:12.164923 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2746a7b-970b-4c82-876d-1d846b6b70ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:12.166684 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2746a7b-970b-4c82-876d-1d846b6b70ea admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:12.167298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2746a7b-970b-4c82-876d-1d846b6b70ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:12.167989 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2746a7b-970b-4c82-876d-1d846b6b70ea admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:12.169319 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2746a7b-970b-4c82-876d-1d846b6b70ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:12.175815 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2746a7b-970b-4c82-876d-1d846b6b70ea admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:12.181230 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 624/1272] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:12 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:12.181230 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9b1585-be68-4629-ba32-8cd8090bb44c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:12.208846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 649/1273] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:12 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:12.208846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2746a7b-970b-4c82-876d-1d846b6b70ea admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:12.210746 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9b1585-be68-4629-ba32-8cd8090bb44c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:12.218009 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8dee78-4f15-43e1-80a0-f9dc5cbb1a3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:12.219364 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8dee78-4f15-43e1-80a0-f9dc5cbb1a3b admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:12.219812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8dee78-4f15-43e1-80a0-f9dc5cbb1a3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:12.220254 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8dee78-4f15-43e1-80a0-f9dc5cbb1a3b admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:12.221372 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8dee78-4f15-43e1-80a0-f9dc5cbb1a3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:12.221757 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2746a7b-970b-4c82-876d-1d846b6b70ea admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:12.223710 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8dee78-4f15-43e1-80a0-f9dc5cbb1a3b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:12.228551 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8bb983-17d9-4129-b308-3f125b5ad8b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:12.229415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8bb983-17d9-4129-b308-3f125b5ad8b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:12.229769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8bb983-17d9-4129-b308-3f125b5ad8b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:12.230116 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8bb983-17d9-4129-b308-3f125b5ad8b8 admin admin] PATH_INFO: `/v3/projects/0ebf28d90fe7429d94e65b524ce8d9de/users/2a14964894f1476a9c7125f87711a754/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:12.240234 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 625/1274] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:12 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:12.241797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66294bba-685d-4869-998e-c1464a9fdc77 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:12.242109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66294bba-685d-4869-998e-c1464a9fdc77 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:12.242379 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66294bba-685d-4869-998e-c1464a9fdc77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:12.267879 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8bb983-17d9-4129-b308-3f125b5ad8b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ebf28d90fe7429d94e65b524ce8d9de, user_id=2a14964894f1476a9c7125f87711a754, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:12.275881 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8bb983-17d9-4129-b308-3f125b5ad8b8 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:12.278891 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-66294bba-685d-4869-998e-c1464a9fdc77 None None] MFA Rules not processed for user `ca7148520e6b4c3e8f5400a2d79ebc87`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:12.281811 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-66294bba-685d-4869-998e-c1464a9fdc77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:12.291765 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:12.306360 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 650/1275] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:12 2022] PUT /identity/v3/projects/0ebf28d90fe7429d94e65b524ce8d9de/users/2a14964894f1476a9c7125f87711a754/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:12.307861 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8bb983-17d9-4129-b308-3f125b5ad8b8 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:12.396435 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 626/1276] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:12 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:12.397409 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66294bba-685d-4869-998e-c1464a9fdc77 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:12.416215 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66294bba-685d-4869-998e-c1464a9fdc77 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:12.418166 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8bb983-17d9-4129-b308-3f125b5ad8b8 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:12.423190 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c910e597-a622-4699-8877-cbcc77bb66c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:12.428102 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c910e597-a622-4699-8877-cbcc77bb66c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:12.428377 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c910e597-a622-4699-8877-cbcc77bb66c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:12.428663 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c910e597-a622-4699-8877-cbcc77bb66c4 admin admin] PATH_INFO: `/v3/projects/5462fcbfca114c8885f77f42b479b13d/users/b01ce2d27bdb4c91a3f55c891f949e00/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:12.429360 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a97faf9-0a6b-47f7-bbb3-5f97baa56a9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:12.430915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a97faf9-0a6b-47f7-bbb3-5f97baa56a9e admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:12.433874 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a97faf9-0a6b-47f7-bbb3-5f97baa56a9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:12.434823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a97faf9-0a6b-47f7-bbb3-5f97baa56a9e admin admin] PATH_INFO: `/v3/projects/cca355ef4a294ecd88cc299b7e0ca9d8/users/ccece85a85fb48c4887b52c96c5bc642/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:12.452506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c910e597-a622-4699-8877-cbcc77bb66c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5462fcbfca114c8885f77f42b479b13d, user_id=b01ce2d27bdb4c91a3f55c891f949e00, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:12.455900 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c910e597-a622-4699-8877-cbcc77bb66c4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:12.493005 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 627/1277] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:12 2022] PUT /identity/v3/projects/5462fcbfca114c8885f77f42b479b13d/users/b01ce2d27bdb4c91a3f55c891f949e00/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:12.494078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c910e597-a622-4699-8877-cbcc77bb66c4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:12.501678 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a97faf9-0a6b-47f7-bbb3-5f97baa56a9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=cca355ef4a294ecd88cc299b7e0ca9d8, user_id=ccece85a85fb48c4887b52c96c5bc642, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:12.505496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a97faf9-0a6b-47f7-bbb3-5f97baa56a9e admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:12.527027 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 651/1278] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:12 2022] PUT /identity/v3/projects/cca355ef4a294ecd88cc299b7e0ca9d8/users/ccece85a85fb48c4887b52c96c5bc642/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:12.529047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a97faf9-0a6b-47f7-bbb3-5f97baa56a9e admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:12.631884 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c910e597-a622-4699-8877-cbcc77bb66c4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:12.640130 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c4935f-4f67-43ee-937b-79dc50719b66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:12.640806 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c4935f-4f67-43ee-937b-79dc50719b66 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:12.641963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c4935f-4f67-43ee-937b-79dc50719b66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:12.641963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c4935f-4f67-43ee-937b-79dc50719b66 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:12.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c4935f-4f67-43ee-937b-79dc50719b66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:12.644170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c4935f-4f67-43ee-937b-79dc50719b66 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:12.644556 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a97faf9-0a6b-47f7-bbb3-5f97baa56a9e admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:12.656545 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fc14ac-4245-44af-abbb-051bbded1911 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:12.657757 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fc14ac-4245-44af-abbb-051bbded1911 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:12.661167 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 628/1279] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:12 2022] GET /identity/v3/roles => generated 1420 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:12.661287 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fc14ac-4245-44af-abbb-051bbded1911 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:12.661287 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fc14ac-4245-44af-abbb-051bbded1911 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:12.661500 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c4935f-4f67-43ee-937b-79dc50719b66 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:12.661594 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fc14ac-4245-44af-abbb-051bbded1911 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:12.663522 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fc14ac-4245-44af-abbb-051bbded1911 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:12.674074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c4935f-4f67-43ee-937b-79dc50719b66 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:12.678150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:12.681555 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500453b2-7b4d-4cf0-a230-fe07db7c94bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:12.682323 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500453b2-7b4d-4cf0-a230-fe07db7c94bf admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:12.682777 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500453b2-7b4d-4cf0-a230-fe07db7c94bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:12.683107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500453b2-7b4d-4cf0-a230-fe07db7c94bf admin admin] PATH_INFO: `/v3/projects/fb3c758e27684dcd9265e2d60036d80b/users/466f2f8ddd184a98bdb5e922818905cc/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:12.695889 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 652/1280] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:12 2022] GET /identity/v3/roles => generated 1420 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:12.696444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e45702-2a74-48f4-9123-bf064145b45f None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:12.697125 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e45702-2a74-48f4-9123-bf064145b45f None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:12.698259 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e45702-2a74-48f4-9123-bf064145b45f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:12.709233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500453b2-7b4d-4cf0-a230-fe07db7c94bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb3c758e27684dcd9265e2d60036d80b, user_id=466f2f8ddd184a98bdb5e922818905cc, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:12.712493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500453b2-7b4d-4cf0-a230-fe07db7c94bf admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:12.732652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 629/1281] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:12 2022] PUT /identity/v3/projects/fb3c758e27684dcd9265e2d60036d80b/users/466f2f8ddd184a98bdb5e922818905cc/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:12.739005 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a0cb2c-78c0-46ff-806b-7d03e576c6f0 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:12.739451 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a0cb2c-78c0-46ff-806b-7d03e576c6f0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:12.739877 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a0cb2c-78c0-46ff-806b-7d03e576c6f0 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:12.741644 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 630/1282] 10.222.0.25 () {54 vars in 961 bytes} [Sun Oct 16 00:57:12 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:57:12.743189 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7e8c84-a8b1-4e07-872c-b1528b187d07 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:12.743474 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7e8c84-a8b1-4e07-872c-b1528b187d07 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:12.743749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7e8c84-a8b1-4e07-872c-b1528b187d07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:12.751959 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-89e45702-2a74-48f4-9123-bf064145b45f None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:12.753381 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-89e45702-2a74-48f4-9123-bf064145b45f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:12.785271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-1c7e8c84-a8b1-4e07-872c-b1528b187d07 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:12.787582 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1c7e8c84-a8b1-4e07-872c-b1528b187d07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:12.839905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 653/1283] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:12.841835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a6fe16-efdb-4686-88ee-6c7c35f3797c None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:12.843364 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a6fe16-efdb-4686-88ee-6c7c35f3797c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:12.844074 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a6fe16-efdb-4686-88ee-6c7c35f3797c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:12.888539 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-b8a6fe16-efdb-4686-88ee-6c7c35f3797c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:12.891512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b8a6fe16-efdb-4686-88ee-6c7c35f3797c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:12.907669 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 654/1284] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:12.909363 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a6fe16-efdb-4686-88ee-6c7c35f3797c None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:12.922903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:12.934754 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a6fe16-efdb-4686-88ee-6c7c35f3797c None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:12.947754 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef79e699-e7d2-49c1-bb44-9fe146217af7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:12.954694 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef79e699-e7d2-49c1-bb44-9fe146217af7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:12.955561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef79e699-e7d2-49c1-bb44-9fe146217af7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:12.956400 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef79e699-e7d2-49c1-bb44-9fe146217af7 admin admin] PATH_INFO: `/v3/projects/db213870f6a145cf9405415bdeb0a92b/users/63ebc9b0925a4b078f35f7faf3c5fd84/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:12.959245 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 631/1285] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:12.961145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7e8c84-a8b1-4e07-872c-b1528b187d07 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:12.983841 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7e8c84-a8b1-4e07-872c-b1528b187d07 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:12.994146 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef79e699-e7d2-49c1-bb44-9fe146217af7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db213870f6a145cf9405415bdeb0a92b, user_id=63ebc9b0925a4b078f35f7faf3c5fd84, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:12.997645 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef79e699-e7d2-49c1-bb44-9fe146217af7 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:12.997916 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666d3882-b791-449a-9072-6b8928982ff8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:13.000129 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666d3882-b791-449a-9072-6b8928982ff8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:13.000129 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666d3882-b791-449a-9072-6b8928982ff8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:13.000436 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666d3882-b791-449a-9072-6b8928982ff8 admin admin] PATH_INFO: `/v3/projects/ce82a4cafaac4ccb82a0e07c3212584f/users/4c3248ed5595438b924cea73f22598ad/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:13.020843 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 655/1286] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:12 2022] PUT /identity/v3/projects/db213870f6a145cf9405415bdeb0a92b/users/63ebc9b0925a4b078f35f7faf3c5fd84/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:13.027676 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0f2e50-401c-4a2b-ad3b-2458acd7126b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:13.028409 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0f2e50-401c-4a2b-ad3b-2458acd7126b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:13.028962 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0f2e50-401c-4a2b-ad3b-2458acd7126b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:13.040968 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666d3882-b791-449a-9072-6b8928982ff8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ce82a4cafaac4ccb82a0e07c3212584f, user_id=4c3248ed5595438b924cea73f22598ad, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:13.048308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666d3882-b791-449a-9072-6b8928982ff8 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:13.068591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-5e0f2e50-401c-4a2b-ad3b-2458acd7126b None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:13.071170 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-5e0f2e50-401c-4a2b-ad3b-2458acd7126b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:13.083975 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 632/1287] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:12 2022] PUT /identity/v3/projects/ce82a4cafaac4ccb82a0e07c3212584f/users/4c3248ed5595438b924cea73f22598ad/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:13.087141 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd19a781-87d5-473c-bbb0-7fee51b5a098 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:13.087631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd19a781-87d5-473c-bbb0-7fee51b5a098 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:13.088050 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd19a781-87d5-473c-bbb0-7fee51b5a098 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:13.171419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-fd19a781-87d5-473c-bbb0-7fee51b5a098 None None] MFA Rules not processed for user `4eba26813e154595a12a67fe4be5eeff`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:13.173469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fd19a781-87d5-473c-bbb0-7fee51b5a098 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:13.218069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 656/1288] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:13.224283 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0f2e50-401c-4a2b-ad3b-2458acd7126b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:13.268404 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 633/1289] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Oct 16 00:57:13 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:13.269580 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd19a781-87d5-473c-bbb0-7fee51b5a098 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:13.312279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0f2e50-401c-4a2b-ad3b-2458acd7126b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:13.316844 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fba0335-e180-4968-bb80-c7ad3985ac59 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:13.318137 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fba0335-e180-4968-bb80-c7ad3985ac59 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:13.318657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fba0335-e180-4968-bb80-c7ad3985ac59 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:13.319192 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fba0335-e180-4968-bb80-c7ad3985ac59 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:13.332768 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fba0335-e180-4968-bb80-c7ad3985ac59 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:13.336085 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fba0335-e180-4968-bb80-c7ad3985ac59 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:13.350431 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 657/1290] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:13.351714 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fba0335-e180-4968-bb80-c7ad3985ac59 service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:13.360498 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd19a781-87d5-473c-bbb0-7fee51b5a098 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:13.364014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fba0335-e180-4968-bb80-c7ad3985ac59 service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:13.367638 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492d1975-2e30-4086-8e24-f23f58bbe30e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:13.368756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492d1975-2e30-4086-8e24-f23f58bbe30e service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:13.370582 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492d1975-2e30-4086-8e24-f23f58bbe30e service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:13.370582 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492d1975-2e30-4086-8e24-f23f58bbe30e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:13.371808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d411d4ce-274e-4411-9a33-5f7e351e9e91 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:13.373662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d411d4ce-274e-4411-9a33-5f7e351e9e91 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:13.374765 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d411d4ce-274e-4411-9a33-5f7e351e9e91 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:13.375664 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d411d4ce-274e-4411-9a33-5f7e351e9e91 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:13.390738 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d411d4ce-274e-4411-9a33-5f7e351e9e91 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:13.392706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d411d4ce-274e-4411-9a33-5f7e351e9e91 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:13.400591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:13.409747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 658/1291] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:13.410204 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492d1975-2e30-4086-8e24-f23f58bbe30e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:13.412083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089715de-e546-438d-8446-775823dcf17c None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:13.412664 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089715de-e546-438d-8446-775823dcf17c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:13.412814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492d1975-2e30-4086-8e24-f23f58bbe30e service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:13.414230 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089715de-e546-438d-8446-775823dcf17c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:13.434437 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 634/1292] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:13.435946 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5b3df2-f26c-4f8b-af61-55ab7519f4d0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:13.436373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5b3df2-f26c-4f8b-af61-55ab7519f4d0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:13.436666 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5b3df2-f26c-4f8b-af61-55ab7519f4d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:13.462337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-089715de-e546-438d-8446-775823dcf17c None None] MFA Rules not processed for user `63ebc9b0925a4b078f35f7faf3c5fd84`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:13.463948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-089715de-e546-438d-8446-775823dcf17c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:13.487554 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4f5b3df2-f26c-4f8b-af61-55ab7519f4d0 None None] MFA Rules not processed for user `4c3248ed5595438b924cea73f22598ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:13.489623 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4f5b3df2-f26c-4f8b-af61-55ab7519f4d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:13.588545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:13.675164 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 659/1293] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:13 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:13.676699 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089715de-e546-438d-8446-775823dcf17c None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:13.695605 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 635/1294] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:13 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 260 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:13.697782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5b3df2-f26c-4f8b-af61-55ab7519f4d0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:13.705312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089715de-e546-438d-8446-775823dcf17c None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:13.720003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5c36ef-ff11-421d-a263-07b5bf497152 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:13.720645 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5c36ef-ff11-421d-a263-07b5bf497152 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:13.720923 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5c36ef-ff11-421d-a263-07b5bf497152 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:13.721211 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5c36ef-ff11-421d-a263-07b5bf497152 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:13.744116 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5c36ef-ff11-421d-a263-07b5bf497152 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:13.745553 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5b3df2-f26c-4f8b-af61-55ab7519f4d0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:13.745794 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5c36ef-ff11-421d-a263-07b5bf497152 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:13.761192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8519db81-6558-4300-9e0a-cf3e734c4fa1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:13.762962 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8519db81-6558-4300-9e0a-cf3e734c4fa1 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:13.764013 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8519db81-6558-4300-9e0a-cf3e734c4fa1 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:13.765469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8519db81-6558-4300-9e0a-cf3e734c4fa1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:13.774752 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 660/1295] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:13.775738 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5c36ef-ff11-421d-a263-07b5bf497152 service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:13.809430 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5c36ef-ff11-421d-a263-07b5bf497152 service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:13.823233 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7536ff1d-9e15-4d03-b69c-f4fc8eb6c196 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:13.824274 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7536ff1d-9e15-4d03-b69c-f4fc8eb6c196 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:13.824574 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7536ff1d-9e15-4d03-b69c-f4fc8eb6c196 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:13.824574 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7536ff1d-9e15-4d03-b69c-f4fc8eb6c196 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:13.825214 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7536ff1d-9e15-4d03-b69c-f4fc8eb6c196 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:13.834675 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7536ff1d-9e15-4d03-b69c-f4fc8eb6c196 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:13.911542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 661/1296] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:13 2022] POST /identity/v3/projects => generated 365 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:13.911542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eaffa2-9962-44bb-aa0c-0b835777db3c None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:13.911542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eaffa2-9962-44bb-aa0c-0b835777db3c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:13.911542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eaffa2-9962-44bb-aa0c-0b835777db3c None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:13.918661 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 662/1297] 10.222.0.25 () {54 vars in 961 bytes} [Sun Oct 16 00:57:13 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:57:13.920242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06eaffa2-9962-44bb-aa0c-0b835777db3c None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:14.004121 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8519db81-6558-4300-9e0a-cf3e734c4fa1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:14.013380 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8519db81-6558-4300-9e0a-cf3e734c4fa1 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:14.058225 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 636/1298] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:13 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:14.063464 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af42f787-70d7-4036-bc4e-4ccc57893cd8 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:14.065215 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af42f787-70d7-4036-bc4e-4ccc57893cd8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:14.065664 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af42f787-70d7-4036-bc4e-4ccc57893cd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:14.075756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06eaffa2-9962-44bb-aa0c-0b835777db3c None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:14.083168 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7b4c25-bf07-49a8-a6dd-5512a64b5639 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:14.084013 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7b4c25-bf07-49a8-a6dd-5512a64b5639 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:14.084523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7b4c25-bf07-49a8-a6dd-5512a64b5639 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:14.084900 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7b4c25-bf07-49a8-a6dd-5512a64b5639 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:14.085922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7b4c25-bf07-49a8-a6dd-5512a64b5639 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:14.089763 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7b4c25-bf07-49a8-a6dd-5512a64b5639 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:14.240948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 663/1299] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:13 2022] POST /identity/v3/users => generated 289 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:14.240948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7b4c25-bf07-49a8-a6dd-5512a64b5639 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:14.240948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7b4c25-bf07-49a8-a6dd-5512a64b5639 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:14.240948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f18e07-2e66-4fe4-a4ea-f2690ec29081 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:14.242376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f18e07-2e66-4fe4-a4ea-f2690ec29081 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:14.242376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f18e07-2e66-4fe4-a4ea-f2690ec29081 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:14.242376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f18e07-2e66-4fe4-a4ea-f2690ec29081 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:14.242376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f18e07-2e66-4fe4-a4ea-f2690ec29081 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:14.242376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f18e07-2e66-4fe4-a4ea-f2690ec29081 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:14.243565 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-af42f787-70d7-4036-bc4e-4ccc57893cd8 None None] MFA Rules not processed for user `4eba26813e154595a12a67fe4be5eeff`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:14.243565 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-af42f787-70d7-4036-bc4e-4ccc57893cd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:14.261528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 664/1300] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:14 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:14.265260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f18e07-2e66-4fe4-a4ea-f2690ec29081 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:14.283512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f18e07-2e66-4fe4-a4ea-f2690ec29081 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:14.316854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d04344-c527-4fbe-830c-847da06203b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:14.317695 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d04344-c527-4fbe-830c-847da06203b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:14.318073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d04344-c527-4fbe-830c-847da06203b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:14.318452 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d04344-c527-4fbe-830c-847da06203b9 admin admin] PATH_INFO: `/v3/projects/7d6e0d35b8394e6fb979a7ba3544d31f/users/c421a4762f55475786abf7688aa55d6a/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:14.371952 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d04344-c527-4fbe-830c-847da06203b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7d6e0d35b8394e6fb979a7ba3544d31f, user_id=c421a4762f55475786abf7688aa55d6a, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:14.377346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d04344-c527-4fbe-830c-847da06203b9 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:14.404379 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 665/1301] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:14 2022] PUT /identity/v3/projects/7d6e0d35b8394e6fb979a7ba3544d31f/users/c421a4762f55475786abf7688aa55d6a/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:14.410065 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d04344-c527-4fbe-830c-847da06203b9 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:14.429344 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:14.486996 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 637/1302] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Oct 16 00:57:14 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 428 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:14.503463 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af42f787-70d7-4036-bc4e-4ccc57893cd8 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:14.522110 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af42f787-70d7-4036-bc4e-4ccc57893cd8 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:14.585247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d04344-c527-4fbe-830c-847da06203b9 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:14.606080 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cd594e-226f-4a0c-b598-04290b4ebea7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:14.606942 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cd594e-226f-4a0c-b598-04290b4ebea7 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:14.607210 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cd594e-226f-4a0c-b598-04290b4ebea7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:14.607480 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cd594e-226f-4a0c-b598-04290b4ebea7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:14.611424 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8cd594e-226f-4a0c-b598-04290b4ebea7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:14.616403 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8cd594e-226f-4a0c-b598-04290b4ebea7 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:14.618529 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9572f8-425c-4fe6-8db2-86ed90769a4d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:14.619352 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9572f8-425c-4fe6-8db2-86ed90769a4d service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:14.619702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9572f8-425c-4fe6-8db2-86ed90769a4d service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:14.620048 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9572f8-425c-4fe6-8db2-86ed90769a4d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:14.657573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 666/1303] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:14 2022] GET /identity/v3/roles => generated 1420 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:14.661079 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:14.661599 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cd594e-226f-4a0c-b598-04290b4ebea7 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:14.682726 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cd594e-226f-4a0c-b598-04290b4ebea7 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:14.693884 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc308ae-ab57-4ba4-ae21-a5ca691e7db1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:14.695150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc308ae-ab57-4ba4-ae21-a5ca691e7db1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:14.695560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc308ae-ab57-4ba4-ae21-a5ca691e7db1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:14.695938 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc308ae-ab57-4ba4-ae21-a5ca691e7db1 admin admin] PATH_INFO: `/v3/projects/7d6e0d35b8394e6fb979a7ba3544d31f/users/c421a4762f55475786abf7688aa55d6a/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:14.698204 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc308ae-ab57-4ba4-ae21-a5ca691e7db1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7d6e0d35b8394e6fb979a7ba3544d31f, user_id=c421a4762f55475786abf7688aa55d6a, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:14.703174 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bc308ae-ab57-4ba4-ae21-a5ca691e7db1 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:14.729739 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 667/1304] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:14 2022] PUT /identity/v3/projects/7d6e0d35b8394e6fb979a7ba3544d31f/users/c421a4762f55475786abf7688aa55d6a/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:14.753346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ebd42f-d1a8-4d44-8f83-728af501b0bf None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:14.753649 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ebd42f-d1a8-4d44-8f83-728af501b0bf None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:14.753932 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ebd42f-d1a8-4d44-8f83-728af501b0bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:14.816984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9572f8-425c-4fe6-8db2-86ed90769a4d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:14.818988 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9572f8-425c-4fe6-8db2-86ed90769a4d service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:14.848727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 638/1305] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:14 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:14.851115 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f66a729-7fab-443a-a40c-c53d7d44f40b None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:14.851526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f66a729-7fab-443a-a40c-c53d7d44f40b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:14.851986 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f66a729-7fab-443a-a40c-c53d7d44f40b None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:14.857257 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 639/1306] 10.222.0.25 () {54 vars in 912 bytes} [Sun Oct 16 00:57:14 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:57:14.866563 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-50ebd42f-d1a8-4d44-8f83-728af501b0bf None None] MFA Rules not processed for user `c421a4762f55475786abf7688aa55d6a`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:14.867693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a61fecc-444e-47ce-8d9f-c583b357e46f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:14.868425 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a61fecc-444e-47ce-8d9f-c583b357e46f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:14.868657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-50ebd42f-d1a8-4d44-8f83-728af501b0bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:14.869354 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a61fecc-444e-47ce-8d9f-c583b357e46f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:14.918634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-0a61fecc-444e-47ce-8d9f-c583b357e46f None None] MFA Rules not processed for user `4eba26813e154595a12a67fe4be5eeff`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:14.923856 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0a61fecc-444e-47ce-8d9f-c583b357e46f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:15.058175 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 668/1307] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:14 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:15.067607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ebd42f-d1a8-4d44-8f83-728af501b0bf None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:15.149520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 640/1308] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 16 00:57:14 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 285 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:57:15.153304 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a61fecc-444e-47ce-8d9f-c583b357e46f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:15.239117 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ebd42f-d1a8-4d44-8f83-728af501b0bf None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:15.251586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202b20c0-b98c-4988-8827-478c750eac44 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:15.252768 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202b20c0-b98c-4988-8827-478c750eac44 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:15.255170 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202b20c0-b98c-4988-8827-478c750eac44 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:15.255657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202b20c0-b98c-4988-8827-478c750eac44 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:15.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202b20c0-b98c-4988-8827-478c750eac44 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:15.289372 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202b20c0-b98c-4988-8827-478c750eac44 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:15.318966 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a61fecc-444e-47ce-8d9f-c583b357e46f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:15.319985 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 669/1309] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:15 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:57:15.325123 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28992a39-7b08-4368-bd9b-47b739330b2d None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:15.325521 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28992a39-7b08-4368-bd9b-47b739330b2d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:15.325923 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28992a39-7b08-4368-bd9b-47b739330b2d None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:15.330367 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999bc765-77b4-4b68-b614-0614151967fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:15.331192 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 670/1310] 10.222.0.25 () {54 vars in 912 bytes} [Sun Oct 16 00:57:15 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:57:15.331642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999bc765-77b4-4b68-b614-0614151967fd service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:15.332568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999bc765-77b4-4b68-b614-0614151967fd service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:15.332850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999bc765-77b4-4b68-b614-0614151967fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:15.337207 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20909774-4eb3-41ab-9e1b-8e285fa4275a None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:15.337649 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20909774-4eb3-41ab-9e1b-8e285fa4275a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:15.338031 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20909774-4eb3-41ab-9e1b-8e285fa4275a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:15.358463 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999bc765-77b4-4b68-b614-0614151967fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:15.362311 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999bc765-77b4-4b68-b614-0614151967fd service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:15.411572 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-20909774-4eb3-41ab-9e1b-8e285fa4275a None None] MFA Rules not processed for user `4eba26813e154595a12a67fe4be5eeff`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:15.412967 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-20909774-4eb3-41ab-9e1b-8e285fa4275a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:15.424259 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 641/1311] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 274 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:15.451059 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 671/1312] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 16 00:57:15 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:57:15.519997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999bc765-77b4-4b68-b614-0614151967fd service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:15.532031 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999bc765-77b4-4b68-b614-0614151967fd service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:15.538070 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0c90a3-c1b5-4b12-8c00-c2b207f29322 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:15.538780 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0c90a3-c1b5-4b12-8c00-c2b207f29322 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:15.539229 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0c90a3-c1b5-4b12-8c00-c2b207f29322 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:15.539615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0c90a3-c1b5-4b12-8c00-c2b207f29322 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:15.559142 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20909774-4eb3-41ab-9e1b-8e285fa4275a None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:15.578638 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20909774-4eb3-41ab-9e1b-8e285fa4275a None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:15.587108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0c90a3-c1b5-4b12-8c00-c2b207f29322 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:15.587717 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d02bd5f-5ea4-4bfd-93c9-a23ac0c94fb5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:15.588477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d02bd5f-5ea4-4bfd-93c9-a23ac0c94fb5 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:15.588788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d02bd5f-5ea4-4bfd-93c9-a23ac0c94fb5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:15.589120 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d02bd5f-5ea4-4bfd-93c9-a23ac0c94fb5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:15.593418 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0c90a3-c1b5-4b12-8c00-c2b207f29322 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:15.623356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:15.625664 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 642/1313] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:15.750706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d02bd5f-5ea4-4bfd-93c9-a23ac0c94fb5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:15.756340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d02bd5f-5ea4-4bfd-93c9-a23ac0c94fb5 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:15.807843 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 672/1314] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:15 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:57:16.781995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0c90a3-c1b5-4b12-8c00-c2b207f29322 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:16.913904 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:17.014537 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0c90a3-c1b5-4b12-8c00-c2b207f29322 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:17.031998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7a4fe6-c2e7-4bdb-b068-0efcb096dbce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:17.032860 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7a4fe6-c2e7-4bdb-b068-0efcb096dbce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:17.033428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7a4fe6-c2e7-4bdb-b068-0efcb096dbce admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:17.039172 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7a4fe6-c2e7-4bdb-b068-0efcb096dbce admin admin] PATH_INFO: `/v3/users/ca7148520e6b4c3e8f5400a2d79ebc87` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:17.041901 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7a4fe6-c2e7-4bdb-b068-0efcb096dbce admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca7148520e6b4c3e8f5400a2d79ebc87)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:17.045842 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: WARNING py.warnings [None req-fb7a4fe6-c2e7-4bdb-b068-0efcb096dbce admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 16 00:57:17.045842 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: warnings.warn(msg) Oct 16 00:57:17.045842 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: Oct 16 00:57:17.049591 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7a4fe6-c2e7-4bdb-b068-0efcb096dbce admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:17.226328 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-fb7a4fe6-c2e7-4bdb-b068-0efcb096dbce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca7148520e6b4c3e8f5400a2d79ebc87'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:17.229824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-fb7a4fe6-c2e7-4bdb-b068-0efcb096dbce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca7148520e6b4c3e8f5400a2d79ebc87'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:17.245757 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-fb7a4fe6-c2e7-4bdb-b068-0efcb096dbce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca7148520e6b4c3e8f5400a2d79ebc87'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:17.249427 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-fb7a4fe6-c2e7-4bdb-b068-0efcb096dbce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca7148520e6b4c3e8f5400a2d79ebc87'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:17.313537 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-fb7a4fe6-c2e7-4bdb-b068-0efcb096dbce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca7148520e6b4c3e8f5400a2d79ebc87'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:17.342711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 643/1315] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:57:16 2022] DELETE /identity/v3/users/ca7148520e6b4c3e8f5400a2d79ebc87 => generated 0 bytes in 561 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:17.348506 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1925b6b7-0d0c-48c8-ad70-7b21e0d885c6 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:17.349322 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1925b6b7-0d0c-48c8-ad70-7b21e0d885c6 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:17.349927 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1925b6b7-0d0c-48c8-ad70-7b21e0d885c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:17.379934 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-1925b6b7-0d0c-48c8-ad70-7b21e0d885c6 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:17.382916 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-1925b6b7-0d0c-48c8-ad70-7b21e0d885c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:17.550904 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 673/1316] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:17.627878 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7a4fe6-c2e7-4bdb-b068-0efcb096dbce admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:17.630031 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fb7a4fe6-c2e7-4bdb-b068-0efcb096dbce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:17.832363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7a4fe6-c2e7-4bdb-b068-0efcb096dbce admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:17.854978 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c24f12e-e544-45b5-bd3b-738a490dd62e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:17.855541 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c24f12e-e544-45b5-bd3b-738a490dd62e service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:17.855834 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c24f12e-e544-45b5-bd3b-738a490dd62e service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:17.856106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c24f12e-e544-45b5-bd3b-738a490dd62e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:17.885720 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c24f12e-e544-45b5-bd3b-738a490dd62e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:17.887478 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c24f12e-e544-45b5-bd3b-738a490dd62e service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:17.913169 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 644/1317] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 286 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:57:18.230370 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cf58eb-289a-4217-87b1-30e300e8fbf0 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:18.230759 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cf58eb-289a-4217-87b1-30e300e8fbf0 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:18.231893 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cf58eb-289a-4217-87b1-30e300e8fbf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:18.299106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-14cf58eb-289a-4217-87b1-30e300e8fbf0 None None] MFA Rules not processed for user `2a14964894f1476a9c7125f87711a754`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:18.303235 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-14cf58eb-289a-4217-87b1-30e300e8fbf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:18.582919 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 674/1318] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:18 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 353 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:18.588302 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c24f12e-e544-45b5-bd3b-738a490dd62e service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:18.589506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5c24f12e-e544-45b5-bd3b-738a490dd62e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:18.643943 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c24f12e-e544-45b5-bd3b-738a490dd62e service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:18.682758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4856cc92-90f0-4261-8f09-900e0ffcc6f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:18.688544 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4856cc92-90f0-4261-8f09-900e0ffcc6f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:18.692169 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4856cc92-90f0-4261-8f09-900e0ffcc6f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:18.692782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4856cc92-90f0-4261-8f09-900e0ffcc6f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:18.702493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4856cc92-90f0-4261-8f09-900e0ffcc6f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:18.712066 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4856cc92-90f0-4261-8f09-900e0ffcc6f5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:18.784872 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 645/1319] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:18 2022] POST /identity/v3/projects => generated 401 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:18.791171 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14cf58eb-289a-4217-87b1-30e300e8fbf0 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:18.821177 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14cf58eb-289a-4217-87b1-30e300e8fbf0 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:18.957809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:19.014736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8977e068-f4ed-4794-a042-cfcde4617e5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:19.015420 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8977e068-f4ed-4794-a042-cfcde4617e5e admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:19.015702 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8977e068-f4ed-4794-a042-cfcde4617e5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:19.015979 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8977e068-f4ed-4794-a042-cfcde4617e5e admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:19.016764 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8977e068-f4ed-4794-a042-cfcde4617e5e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:19.019102 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8977e068-f4ed-4794-a042-cfcde4617e5e admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:19.072580 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 675/1320] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:18 2022] POST /identity/v3/users => generated 307 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:19.074239 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4856cc92-90f0-4261-8f09-900e0ffcc6f5 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:19.178243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4856cc92-90f0-4261-8f09-900e0ffcc6f5 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:19.208623 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3b760e-d273-4d6b-a42a-7c310ec3d859 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:19.209723 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3b760e-d273-4d6b-a42a-7c310ec3d859 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:19.211466 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3b760e-d273-4d6b-a42a-7c310ec3d859 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:19.213260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3b760e-d273-4d6b-a42a-7c310ec3d859 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:19.216718 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3b760e-d273-4d6b-a42a-7c310ec3d859 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:19.226126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3b760e-d273-4d6b-a42a-7c310ec3d859 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:19.263380 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 646/1321] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:19 2022] GET /identity/v3/roles => generated 1420 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:19.267157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8977e068-f4ed-4794-a042-cfcde4617e5e admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:19.281877 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8977e068-f4ed-4794-a042-cfcde4617e5e admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:19.300791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4502174a-0809-4575-baed-b2e045d23713 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:19.301513 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4502174a-0809-4575-baed-b2e045d23713 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:19.301924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4502174a-0809-4575-baed-b2e045d23713 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:19.302211 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4502174a-0809-4575-baed-b2e045d23713 admin admin] PATH_INFO: `/v3/projects/1daba3a4330d42c8a14ccdf105b6738c/users/e5773c56e4954981be04d587c61be9d6/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:19.333542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4502174a-0809-4575-baed-b2e045d23713 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1daba3a4330d42c8a14ccdf105b6738c, user_id=e5773c56e4954981be04d587c61be9d6, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:19.336565 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4502174a-0809-4575-baed-b2e045d23713 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:19.378182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 676/1322] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:19 2022] PUT /identity/v3/projects/1daba3a4330d42c8a14ccdf105b6738c/users/e5773c56e4954981be04d587c61be9d6/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:57:19.382793 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3b760e-d273-4d6b-a42a-7c310ec3d859 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:19.408502 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3b760e-d273-4d6b-a42a-7c310ec3d859 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:19.449900 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:19.580732 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5434de-e88e-4adf-bd39-b27c365dfce7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:19.581400 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5434de-e88e-4adf-bd39-b27c365dfce7 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:19.581692 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5434de-e88e-4adf-bd39-b27c365dfce7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:19.581963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5434de-e88e-4adf-bd39-b27c365dfce7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:19.582716 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5434de-e88e-4adf-bd39-b27c365dfce7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:19.595470 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5434de-e88e-4adf-bd39-b27c365dfce7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:19.618785 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 647/1323] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:19 2022] GET /identity/v3/roles => generated 1420 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 00:57:19.623895 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4502174a-0809-4575-baed-b2e045d23713 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:19.646086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4502174a-0809-4575-baed-b2e045d23713 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:19.704483 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adace67c-cff7-45e2-94b9-5a125b2a2ab1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:19.705736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adace67c-cff7-45e2-94b9-5a125b2a2ab1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:19.709156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adace67c-cff7-45e2-94b9-5a125b2a2ab1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:19.712041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adace67c-cff7-45e2-94b9-5a125b2a2ab1 admin admin] PATH_INFO: `/v3/projects/1daba3a4330d42c8a14ccdf105b6738c/users/e5773c56e4954981be04d587c61be9d6/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:19.729014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adace67c-cff7-45e2-94b9-5a125b2a2ab1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1daba3a4330d42c8a14ccdf105b6738c, user_id=e5773c56e4954981be04d587c61be9d6, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:19.731834 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adace67c-cff7-45e2-94b9-5a125b2a2ab1 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:19.755523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 677/1324] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:19 2022] PUT /identity/v3/projects/1daba3a4330d42c8a14ccdf105b6738c/users/e5773c56e4954981be04d587c61be9d6/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:21.001145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5434de-e88e-4adf-bd39-b27c365dfce7 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:21.003426 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1e5434de-e88e-4adf-bd39-b27c365dfce7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:21.037370 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5434de-e88e-4adf-bd39-b27c365dfce7 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:21.098389 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adace67c-cff7-45e2-94b9-5a125b2a2ab1 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:21.100416 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-adace67c-cff7-45e2-94b9-5a125b2a2ab1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:21.245363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc65042-f134-47f3-9bee-4565d02fbd49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:21.246076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc65042-f134-47f3-9bee-4565d02fbd49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:21.246357 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc65042-f134-47f3-9bee-4565d02fbd49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:21.246646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc65042-f134-47f3-9bee-4565d02fbd49 admin admin] PATH_INFO: `/v3/users/63ebc9b0925a4b078f35f7faf3c5fd84` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:21.252796 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc65042-f134-47f3-9bee-4565d02fbd49 admin admin] RBAC: Authorizing `identity:delete_user(user_id=63ebc9b0925a4b078f35f7faf3c5fd84)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:21.258741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc65042-f134-47f3-9bee-4565d02fbd49 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:21.325334 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adace67c-cff7-45e2-94b9-5a125b2a2ab1 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:21.337478 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11084e4-7795-4422-932b-cf6228bc88e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:21.338625 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11084e4-7795-4422-932b-cf6228bc88e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:21.339156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11084e4-7795-4422-932b-cf6228bc88e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:21.339707 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11084e4-7795-4422-932b-cf6228bc88e5 admin admin] PATH_INFO: `/v3/projects/0d378a01be2044faa872ca43f816dbd9` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:21.347056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11084e4-7795-4422-932b-cf6228bc88e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0d378a01be2044faa872ca43f816dbd9)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:21.348881 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: WARNING py.warnings [None req-d11084e4-7795-4422-932b-cf6228bc88e5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 16 00:57:21.348881 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: warnings.warn(msg) Oct 16 00:57:21.348881 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: Oct 16 00:57:21.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11084e4-7795-4422-932b-cf6228bc88e5 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:21.388338 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2cc65042-f134-47f3-9bee-4565d02fbd49 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '63ebc9b0925a4b078f35f7faf3c5fd84'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:21.391554 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2cc65042-f134-47f3-9bee-4565d02fbd49 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '63ebc9b0925a4b078f35f7faf3c5fd84'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:21.412880 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2cc65042-f134-47f3-9bee-4565d02fbd49 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '63ebc9b0925a4b078f35f7faf3c5fd84'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:21.414611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2cc65042-f134-47f3-9bee-4565d02fbd49 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '63ebc9b0925a4b078f35f7faf3c5fd84'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:21.463354 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2cc65042-f134-47f3-9bee-4565d02fbd49 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '63ebc9b0925a4b078f35f7faf3c5fd84'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:21.509064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 648/1325] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:57:20 2022] DELETE /identity/v3/users/63ebc9b0925a4b078f35f7faf3c5fd84 => generated 0 bytes in 509 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:21.514559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd23e1c0-a734-457d-a16d-1b94178d42af None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:21.515162 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd23e1c0-a734-457d-a16d-1b94178d42af None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:21.515627 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd23e1c0-a734-457d-a16d-1b94178d42af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:21.521580 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d11084e4-7795-4422-932b-cf6228bc88e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0d378a01be2044faa872ca43f816dbd9', 'request_id': 'req-d11084e4-7795-4422-932b-cf6228bc88e5'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:21.538410 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d11084e4-7795-4422-932b-cf6228bc88e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0d378a01be2044faa872ca43f816dbd9', 'request_id': 'req-d11084e4-7795-4422-932b-cf6228bc88e5'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:21.559980 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d11084e4-7795-4422-932b-cf6228bc88e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0d378a01be2044faa872ca43f816dbd9', 'request_id': 'req-d11084e4-7795-4422-932b-cf6228bc88e5'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:21.568516 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-bd23e1c0-a734-457d-a16d-1b94178d42af None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:21.571250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bd23e1c0-a734-457d-a16d-1b94178d42af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:21.622401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d11084e4-7795-4422-932b-cf6228bc88e5 admin admin] The token cache is being invalidate because project 0d378a01be2044faa872ca43f816dbd9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:57:21.622851 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d11084e4-7795-4422-932b-cf6228bc88e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:21.625650 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d11084e4-7795-4422-932b-cf6228bc88e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:21.627818 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 678/1326] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:57:21 2022] DELETE /identity/v3/projects/0d378a01be2044faa872ca43f816dbd9 => generated 0 bytes in 530 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:57:21.633059 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e1c486-e611-41f6-be73-bc8c2e341f0a None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:21.633365 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e1c486-e611-41f6-be73-bc8c2e341f0a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:21.633627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e1c486-e611-41f6-be73-bc8c2e341f0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:21.655122 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:21.677394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-b4e1c486-e611-41f6-be73-bc8c2e341f0a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:21.678447 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b4e1c486-e611-41f6-be73-bc8c2e341f0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:21.721314 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=87,keystone:INSERT=6,keystone:DELETE=7 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:21.731537 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 649/1327] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:57:21.762160 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd23e1c0-a734-457d-a16d-1b94178d42af None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:21.765136 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bd23e1c0-a734-457d-a16d-1b94178d42af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:21.807087 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 679/1328] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:21.808855 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b97ea7-7839-4b6a-8b2d-b08cc2eb24ab None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:21.809152 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b97ea7-7839-4b6a-8b2d-b08cc2eb24ab None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:21.809434 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b97ea7-7839-4b6a-8b2d-b08cc2eb24ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:21.845377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-39b97ea7-7839-4b6a-8b2d-b08cc2eb24ab None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:21.847735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-39b97ea7-7839-4b6a-8b2d-b08cc2eb24ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:21.864360 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 680/1329] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:21.865719 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b97ea7-7839-4b6a-8b2d-b08cc2eb24ab None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:21.880489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b97ea7-7839-4b6a-8b2d-b08cc2eb24ab None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:21.887483 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff7aa14-18fb-46d6-934e-4f48b3e8f39c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:21.890515 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff7aa14-18fb-46d6-934e-4f48b3e8f39c admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:21.891332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff7aa14-18fb-46d6-934e-4f48b3e8f39c admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:21.892154 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff7aa14-18fb-46d6-934e-4f48b3e8f39c admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:21.894654 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff7aa14-18fb-46d6-934e-4f48b3e8f39c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:57:21.895912 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff7aa14-18fb-46d6-934e-4f48b3e8f39c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:21.898263 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff7aa14-18fb-46d6-934e-4f48b3e8f39c admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:21.918439 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd23e1c0-a734-457d-a16d-1b94178d42af None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:21.929368 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 681/1330] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:57:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:57:21.932894 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff7aa14-18fb-46d6-934e-4f48b3e8f39c admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:21.939194 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f649d18d-6c23-4d17-8dd9-f694949a4109 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:21.940800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f649d18d-6c23-4d17-8dd9-f694949a4109 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:21.941924 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f649d18d-6c23-4d17-8dd9-f694949a4109 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:21.944196 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f649d18d-6c23-4d17-8dd9-f694949a4109 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:21.964174 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff7aa14-18fb-46d6-934e-4f48b3e8f39c admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:21.971509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f649d18d-6c23-4d17-8dd9-f694949a4109 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:21.975049 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f649d18d-6c23-4d17-8dd9-f694949a4109 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:21.981677 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2adcd798-1edb-4157-a060-c3c22005c604 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:21.982582 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adcd798-1edb-4157-a060-c3c22005c604 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:21.982845 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adcd798-1edb-4157-a060-c3c22005c604 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:21.984534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2adcd798-1edb-4157-a060-c3c22005c604 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:21.985412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2adcd798-1edb-4157-a060-c3c22005c604 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:21.994393 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2adcd798-1edb-4157-a060-c3c22005c604 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:22.010918 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 650/1331] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 249 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:22.052422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 682/1332] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:21 2022] POST /identity/v3/projects => generated 379 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:22.057127 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f649d18d-6c23-4d17-8dd9-f694949a4109 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:22.072356 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f649d18d-6c23-4d17-8dd9-f694949a4109 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:22.222192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e18309c-cc39-4a2a-a6bf-f6cb036f4ff9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:22.225727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e18309c-cc39-4a2a-a6bf-f6cb036f4ff9 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:22.226210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e18309c-cc39-4a2a-a6bf-f6cb036f4ff9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:22.226662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e18309c-cc39-4a2a-a6bf-f6cb036f4ff9 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:22.227860 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e18309c-cc39-4a2a-a6bf-f6cb036f4ff9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:22.231894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e18309c-cc39-4a2a-a6bf-f6cb036f4ff9 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:22.284131 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2adcd798-1edb-4157-a060-c3c22005c604 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:22.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 651/1333] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:22 2022] POST /identity/v3/users => generated 290 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:22.303042 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2adcd798-1edb-4157-a060-c3c22005c604 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:22.330907 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a83e40-9d3d-4ba6-8c74-298efd12d373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:22.331754 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a83e40-9d3d-4ba6-8c74-298efd12d373 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:22.332860 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a83e40-9d3d-4ba6-8c74-298efd12d373 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:22.333148 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a83e40-9d3d-4ba6-8c74-298efd12d373 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:22.338055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a83e40-9d3d-4ba6-8c74-298efd12d373 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:22.343735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a83e40-9d3d-4ba6-8c74-298efd12d373 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:22.364689 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 683/1334] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:22 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 00:57:22.368957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e18309c-cc39-4a2a-a6bf-f6cb036f4ff9 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:22.389774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e18309c-cc39-4a2a-a6bf-f6cb036f4ff9 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:22.406130 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c00e835-9b56-4943-b6de-6e8f7a062e47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:22.406775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c00e835-9b56-4943-b6de-6e8f7a062e47 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:22.407092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c00e835-9b56-4943-b6de-6e8f7a062e47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:22.407921 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c00e835-9b56-4943-b6de-6e8f7a062e47 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:22.408510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c00e835-9b56-4943-b6de-6e8f7a062e47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:22.410793 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c00e835-9b56-4943-b6de-6e8f7a062e47 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:22.431126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 652/1335] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:22 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:22.435131 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a83e40-9d3d-4ba6-8c74-298efd12d373 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:22.459899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a83e40-9d3d-4ba6-8c74-298efd12d373 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:22.472525 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee949806-2d3c-438e-85bc-91373ffecd2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:22.473488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee949806-2d3c-438e-85bc-91373ffecd2c admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:22.473680 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee949806-2d3c-438e-85bc-91373ffecd2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:22.474047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee949806-2d3c-438e-85bc-91373ffecd2c admin admin] PATH_INFO: `/v3/projects/e0fab4fb42b54a239f72b2696485c3b7/users/cae4e8c5ef13480ab1939587a4fe7d6a/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:22.540053 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee949806-2d3c-438e-85bc-91373ffecd2c admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0fab4fb42b54a239f72b2696485c3b7, user_id=cae4e8c5ef13480ab1939587a4fe7d6a, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:22.557857 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee949806-2d3c-438e-85bc-91373ffecd2c admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:22.615688 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 684/1336] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:22 2022] PUT /identity/v3/projects/e0fab4fb42b54a239f72b2696485c3b7/users/cae4e8c5ef13480ab1939587a4fe7d6a/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:57:22.629566 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42de696f-7ead-4fcf-9d2a-0741d4a40202 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:22.631447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42de696f-7ead-4fcf-9d2a-0741d4a40202 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:22.632282 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42de696f-7ead-4fcf-9d2a-0741d4a40202 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:22.780063 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-42de696f-7ead-4fcf-9d2a-0741d4a40202 None None] MFA Rules not processed for user `cae4e8c5ef13480ab1939587a4fe7d6a`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:22.792061 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-42de696f-7ead-4fcf-9d2a-0741d4a40202 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:22.957177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:22.990133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 653/1337] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:22 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 362 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:22.994681 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee949806-2d3c-438e-85bc-91373ffecd2c admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:23.178835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee949806-2d3c-438e-85bc-91373ffecd2c admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:23.193064 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4be5166-52b7-4c3a-a965-158a03f1f8d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:23.195270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4be5166-52b7-4c3a-a965-158a03f1f8d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:23.195663 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4be5166-52b7-4c3a-a965-158a03f1f8d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:23.197036 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4be5166-52b7-4c3a-a965-158a03f1f8d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:23.197739 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4be5166-52b7-4c3a-a965-158a03f1f8d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:23.205019 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4be5166-52b7-4c3a-a965-158a03f1f8d2 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:23.275414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 685/1338] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:22 2022] POST /identity/v3/projects => generated 379 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:23.277552 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42de696f-7ead-4fcf-9d2a-0741d4a40202 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:23.316860 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42de696f-7ead-4fcf-9d2a-0741d4a40202 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:23.487240 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ba8d1e-1a87-400b-b80c-a9fcb996e4c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:23.488467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ba8d1e-1a87-400b-b80c-a9fcb996e4c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:23.488911 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ba8d1e-1a87-400b-b80c-a9fcb996e4c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:23.489493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ba8d1e-1a87-400b-b80c-a9fcb996e4c1 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:23.490345 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ba8d1e-1a87-400b-b80c-a9fcb996e4c1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:23.492962 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ba8d1e-1a87-400b-b80c-a9fcb996e4c1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:23.536890 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 654/1339] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:23 2022] POST /identity/v3/users => generated 296 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:23.540334 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4be5166-52b7-4c3a-a965-158a03f1f8d2 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:23.554682 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4be5166-52b7-4c3a-a965-158a03f1f8d2 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:23.567916 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f09c85-323d-456c-89ea-296cf11c5d04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:23.568698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f09c85-323d-456c-89ea-296cf11c5d04 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:23.569137 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f09c85-323d-456c-89ea-296cf11c5d04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:23.569752 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f09c85-323d-456c-89ea-296cf11c5d04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:23.570589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f09c85-323d-456c-89ea-296cf11c5d04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:23.572557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f09c85-323d-456c-89ea-296cf11c5d04 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:23.599885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 686/1340] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:23 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 00:57:23.604847 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ba8d1e-1a87-400b-b80c-a9fcb996e4c1 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:23.620904 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ba8d1e-1a87-400b-b80c-a9fcb996e4c1 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:23.636500 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a76d3c-7f69-4e80-9289-ce6988a45205 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:23.637547 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a76d3c-7f69-4e80-9289-ce6988a45205 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:23.637847 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a76d3c-7f69-4e80-9289-ce6988a45205 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:23.638183 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a76d3c-7f69-4e80-9289-ce6988a45205 admin admin] PATH_INFO: `/v3/projects/64bd21611c864051a08b910c68d58539/users/35966572a4b545a5871996616eb4b553/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:23.688181 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a76d3c-7f69-4e80-9289-ce6988a45205 admin admin] RBAC: Authorizing `identity:create_grant(project_id=64bd21611c864051a08b910c68d58539, user_id=35966572a4b545a5871996616eb4b553, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:23.705375 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a76d3c-7f69-4e80-9289-ce6988a45205 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:23.782195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 655/1341] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:23 2022] PUT /identity/v3/projects/64bd21611c864051a08b910c68d58539/users/35966572a4b545a5871996616eb4b553/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:23.789705 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f09c85-323d-456c-89ea-296cf11c5d04 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:23.809456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f09c85-323d-456c-89ea-296cf11c5d04 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:23.870260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:23.914585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a76d3c-7f69-4e80-9289-ce6988a45205 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:23.914585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-89a76d3c-7f69-4e80-9289-ce6988a45205 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:24.067726 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f7b4aa-657b-41ab-9913-c53e3a3b9511 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:24.068805 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f7b4aa-657b-41ab-9913-c53e3a3b9511 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:24.069282 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f7b4aa-657b-41ab-9913-c53e3a3b9511 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:24.069753 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f7b4aa-657b-41ab-9913-c53e3a3b9511 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:24.082601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f7b4aa-657b-41ab-9913-c53e3a3b9511 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:24.095022 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f7b4aa-657b-41ab-9913-c53e3a3b9511 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:24.193993 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a76d3c-7f69-4e80-9289-ce6988a45205 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:24.214337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:24.215734 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:24.215734 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:24.216022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17 admin admin] PATH_INFO: `/v3/projects/db213870f6a145cf9405415bdeb0a92b` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:24.218778 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17 admin admin] RBAC: Authorizing `identity:delete_project(project_id=db213870f6a145cf9405415bdeb0a92b)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:24.224382 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:24.231241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 687/1342] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:23 2022] GET /identity/v3/roles => generated 1420 bytes in 437 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:24.233361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84ed82e-4c91-493a-b842-4c20dd78e1ac None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:24.233584 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84ed82e-4c91-493a-b842-4c20dd78e1ac None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:24.236458 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84ed82e-4c91-493a-b842-4c20dd78e1ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:24.365346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-d84ed82e-4c91-493a-b842-4c20dd78e1ac None None] MFA Rules not processed for user `3aa7409c9f6f46f78164818b6f6927e4`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:24.366664 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d84ed82e-4c91-493a-b842-4c20dd78e1ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:24.403448 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db213870f6a145cf9405415bdeb0a92b', 'request_id': 'req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:24.433019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:24.449799 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db213870f6a145cf9405415bdeb0a92b', 'request_id': 'req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:24.515957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db213870f6a145cf9405415bdeb0a92b', 'request_id': 'req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:24.576177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17 admin admin] The token cache is being invalidate because project db213870f6a145cf9405415bdeb0a92b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:57:24.578191 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:24.597633 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:24.602161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 656/1343] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:57:23 2022] DELETE /identity/v3/projects/db213870f6a145cf9405415bdeb0a92b => generated 0 bytes in 699 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:57:24.604183 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:24.606850 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 688/1344] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:24 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 380 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:24.608831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1ce025-082d-4b07-8704-93a3d76c05fd None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:24.609234 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1ce025-082d-4b07-8704-93a3d76c05fd None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:24.609516 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1ce025-082d-4b07-8704-93a3d76c05fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:24.624742 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:24.683339 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-8a1ce025-082d-4b07-8704-93a3d76c05fd None None] MFA Rules not processed for user `e5773c56e4954981be04d587c61be9d6`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:24.694949 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8a1ce025-082d-4b07-8704-93a3d76c05fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:24.958809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 689/1345] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:24 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 351 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:24.960232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301b2194-e6c3-4793-9ade-b2f85e1a3169 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:24.960523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301b2194-e6c3-4793-9ade-b2f85e1a3169 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:24.960804 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301b2194-e6c3-4793-9ade-b2f85e1a3169 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:25.019318 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8023bc-51c6-4a75-bc76-178bc6d2fd17 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:25.039380 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-301b2194-e6c3-4793-9ade-b2f85e1a3169 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:25.040388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-301b2194-e6c3-4793-9ade-b2f85e1a3169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:25.041391 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d56101-7c8e-4fe5-bde2-de0dba8ad574 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:25.042703 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d56101-7c8e-4fe5-bde2-de0dba8ad574 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:25.045562 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d56101-7c8e-4fe5-bde2-de0dba8ad574 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:25.047619 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d56101-7c8e-4fe5-bde2-de0dba8ad574 admin admin] PATH_INFO: `/v3/projects/64bd21611c864051a08b910c68d58539/users/35966572a4b545a5871996616eb4b553/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:25.058336 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d56101-7c8e-4fe5-bde2-de0dba8ad574 admin admin] RBAC: Authorizing `identity:create_grant(project_id=64bd21611c864051a08b910c68d58539, user_id=35966572a4b545a5871996616eb4b553, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:25.077315 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d56101-7c8e-4fe5-bde2-de0dba8ad574 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:25.083465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 690/1346] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:25.086642 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1b5734-a85f-4a31-9873-b09f2de108e7 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:25.089421 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1b5734-a85f-4a31-9873-b09f2de108e7 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:25.092693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1b5734-a85f-4a31-9873-b09f2de108e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:25.142912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 657/1347] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:24 2022] PUT /identity/v3/projects/64bd21611c864051a08b910c68d58539/users/35966572a4b545a5871996616eb4b553/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 531 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:25.142912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8684e7a-b2cb-4f45-af91-33d287a911c0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:25.142912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8684e7a-b2cb-4f45-af91-33d287a911c0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:25.142912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8684e7a-b2cb-4f45-af91-33d287a911c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:25.160644 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-4e1b5734-a85f-4a31-9873-b09f2de108e7 None None] MFA Rules not processed for user `3aa7409c9f6f46f78164818b6f6927e4`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:25.160644 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4e1b5734-a85f-4a31-9873-b09f2de108e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:25.213016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:25.242188 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c8684e7a-b2cb-4f45-af91-33d287a911c0 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:25.249274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c8684e7a-b2cb-4f45-af91-33d287a911c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:25.322123 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 691/1348] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:25 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:25.324715 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13adc9b-cc4c-405b-83db-db8bd4f7af9b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:25.325034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13adc9b-cc4c-405b-83db-db8bd4f7af9b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:25.325321 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13adc9b-cc4c-405b-83db-db8bd4f7af9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:25.493309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-a13adc9b-cc4c-405b-83db-db8bd4f7af9b None None] MFA Rules not processed for user `35966572a4b545a5871996616eb4b553`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:25.494904 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-a13adc9b-cc4c-405b-83db-db8bd4f7af9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:25.575009 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 658/1349] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 439 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:25.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8684e7a-b2cb-4f45-af91-33d287a911c0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:25.583807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c8684e7a-b2cb-4f45-af91-33d287a911c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:25.788809 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8684e7a-b2cb-4f45-af91-33d287a911c0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:25.800262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7e2324-ef05-4795-a075-04bbedd0fbca service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:25.802972 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7e2324-ef05-4795-a075-04bbedd0fbca service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:25.803480 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7e2324-ef05-4795-a075-04bbedd0fbca service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:25.804002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7e2324-ef05-4795-a075-04bbedd0fbca service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:25.806734 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fc7e2324-ef05-4795-a075-04bbedd0fbca service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:25.819741 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 692/1350] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:25 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 497 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:25.821819 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13adc9b-cc4c-405b-83db-db8bd4f7af9b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:25.848135 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7e2324-ef05-4795-a075-04bbedd0fbca service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:25.850401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7e2324-ef05-4795-a075-04bbedd0fbca service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:25.865198 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13adc9b-cc4c-405b-83db-db8bd4f7af9b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:25.878248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 659/1351] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:25 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:25.882678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7e2324-ef05-4795-a075-04bbedd0fbca service nova] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:25.885617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fc7e2324-ef05-4795-a075-04bbedd0fbca service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:25.907231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ca7a74-817d-4d98-a78e-ffefe602cbb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:25.907885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ca7a74-817d-4d98-a78e-ffefe602cbb9 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:25.908170 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ca7a74-817d-4d98-a78e-ffefe602cbb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:25.908446 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ca7a74-817d-4d98-a78e-ffefe602cbb9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:25.909296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ca7a74-817d-4d98-a78e-ffefe602cbb9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:57:25.909616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ca7a74-817d-4d98-a78e-ffefe602cbb9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:25.911396 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ca7a74-817d-4d98-a78e-ffefe602cbb9 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:25.915828 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:25.945377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 693/1352] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:57:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:57:25.946355 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ca7a74-817d-4d98-a78e-ffefe602cbb9 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:25.975289 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ca7a74-817d-4d98-a78e-ffefe602cbb9 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:25.989549 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7d0fa4-90dd-41df-906b-88bf2742cd0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:25.990108 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7d0fa4-90dd-41df-906b-88bf2742cd0a admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:25.990396 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7d0fa4-90dd-41df-906b-88bf2742cd0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:25.991202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7d0fa4-90dd-41df-906b-88bf2742cd0a admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:25.991891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7d0fa4-90dd-41df-906b-88bf2742cd0a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:25.993924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7d0fa4-90dd-41df-906b-88bf2742cd0a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:26.064166 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 694/1353] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:25 2022] POST /identity/v3/projects => generated 357 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:26.065167 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7d0fa4-90dd-41df-906b-88bf2742cd0a admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:26.185496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7e2324-ef05-4795-a075-04bbedd0fbca service nova] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:26.247098 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7d0fa4-90dd-41df-906b-88bf2742cd0a admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:26.269556 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b31c166-6085-47ac-9a38-ea9e70951dd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:26.270190 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b31c166-6085-47ac-9a38-ea9e70951dd3 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:26.270480 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b31c166-6085-47ac-9a38-ea9e70951dd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:26.270791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b31c166-6085-47ac-9a38-ea9e70951dd3 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:26.272027 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b31c166-6085-47ac-9a38-ea9e70951dd3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:26.273497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b31c166-6085-47ac-9a38-ea9e70951dd3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:26.323548 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 695/1354] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:26 2022] POST /identity/v3/users => generated 279 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:26.333076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b31c166-6085-47ac-9a38-ea9e70951dd3 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:26.364486 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b31c166-6085-47ac-9a38-ea9e70951dd3 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:26.382639 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae0694b-d436-45b9-be7f-147bb4e6251c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:26.383719 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae0694b-d436-45b9-be7f-147bb4e6251c admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:26.386208 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae0694b-d436-45b9-be7f-147bb4e6251c admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:26.386744 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae0694b-d436-45b9-be7f-147bb4e6251c admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:26.387663 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae0694b-d436-45b9-be7f-147bb4e6251c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:26.393760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae0694b-d436-45b9-be7f-147bb4e6251c admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:26.420369 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:26.432755 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e85fc3-5798-4920-b111-e67ebd476568 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:26.433860 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e85fc3-5798-4920-b111-e67ebd476568 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:26.434375 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e85fc3-5798-4920-b111-e67ebd476568 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:26.434890 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e85fc3-5798-4920-b111-e67ebd476568 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:26.448585 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 696/1355] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:26 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:26.456489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae0694b-d436-45b9-be7f-147bb4e6251c admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:26.477658 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae0694b-d436-45b9-be7f-147bb4e6251c admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:26.495147 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2679048e-3651-436e-bb41-ecd21e627ae4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:26.496043 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2679048e-3651-436e-bb41-ecd21e627ae4 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:26.496410 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2679048e-3651-436e-bb41-ecd21e627ae4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:26.496584 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2679048e-3651-436e-bb41-ecd21e627ae4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:26.497236 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2679048e-3651-436e-bb41-ecd21e627ae4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:26.501227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2679048e-3651-436e-bb41-ecd21e627ae4 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:26.528200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 697/1356] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:26 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:26.533031 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2679048e-3651-436e-bb41-ecd21e627ae4 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:26.568467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2679048e-3651-436e-bb41-ecd21e627ae4 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:26.601208 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832965a4-9170-4044-8470-22226248f5a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:26.603285 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832965a4-9170-4044-8470-22226248f5a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:26.603564 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832965a4-9170-4044-8470-22226248f5a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:26.605017 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832965a4-9170-4044-8470-22226248f5a1 admin admin] PATH_INFO: `/v3/projects/42941d5f4dae45c6b1cfa5a803a943d6/users/193e6edd8c334e428fe5aa8235567b80/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:26.669297 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e85fc3-5798-4920-b111-e67ebd476568 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:26.673494 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e85fc3-5798-4920-b111-e67ebd476568 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:26.687950 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832965a4-9170-4044-8470-22226248f5a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=42941d5f4dae45c6b1cfa5a803a943d6, user_id=193e6edd8c334e428fe5aa8235567b80, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:26.709498 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832965a4-9170-4044-8470-22226248f5a1 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:26.737131 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 660/1357] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:25 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 857 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:26.751734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 698/1358] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:26 2022] PUT /identity/v3/projects/42941d5f4dae45c6b1cfa5a803a943d6/users/193e6edd8c334e428fe5aa8235567b80/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:26.756961 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef2c89b-1c9b-4c01-88e3-ebc7b7112ae4 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:26.758215 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef2c89b-1c9b-4c01-88e3-ebc7b7112ae4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:26.758723 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef2c89b-1c9b-4c01-88e3-ebc7b7112ae4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:26.877315 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3ef2c89b-1c9b-4c01-88e3-ebc7b7112ae4 None None] MFA Rules not processed for user `193e6edd8c334e428fe5aa8235567b80`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:26.882381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3ef2c89b-1c9b-4c01-88e3-ebc7b7112ae4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:27.001320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832965a4-9170-4044-8470-22226248f5a1 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:27.073258 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:27.084519 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 661/1359] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:26 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 328 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:27.089069 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef2c89b-1c9b-4c01-88e3-ebc7b7112ae4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:27.192035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832965a4-9170-4044-8470-22226248f5a1 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:27.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb683a3-d132-4d2d-a217-531e1a7cc8f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:27.209603 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb683a3-d132-4d2d-a217-531e1a7cc8f6 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:27.210955 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb683a3-d132-4d2d-a217-531e1a7cc8f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:27.210955 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb683a3-d132-4d2d-a217-531e1a7cc8f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:27.228736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-fcb683a3-d132-4d2d-a217-531e1a7cc8f6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:27.236105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef2c89b-1c9b-4c01-88e3-ebc7b7112ae4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:27.244307 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448d89e8-9e6a-4864-af8a-c06bdec22edc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:27.245412 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448d89e8-9e6a-4864-af8a-c06bdec22edc admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:27.246460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448d89e8-9e6a-4864-af8a-c06bdec22edc admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:27.246951 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448d89e8-9e6a-4864-af8a-c06bdec22edc admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:27.247834 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448d89e8-9e6a-4864-af8a-c06bdec22edc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:27.250517 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448d89e8-9e6a-4864-af8a-c06bdec22edc admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:27.321604 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 662/1360] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:27 2022] POST /identity/v3/projects => generated 357 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:57:27.322605 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448d89e8-9e6a-4864-af8a-c06bdec22edc admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:27.483754 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb683a3-d132-4d2d-a217-531e1a7cc8f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:27.489667 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb683a3-d132-4d2d-a217-531e1a7cc8f6 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:27.502897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448d89e8-9e6a-4864-af8a-c06bdec22edc admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:27.511717 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 699/1361] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:27 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 511 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:27.514191 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3babee14-98c3-4e23-9e8c-c814c3f17dc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:27.515242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3babee14-98c3-4e23-9e8c-c814c3f17dc7 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:27.515528 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3babee14-98c3-4e23-9e8c-c814c3f17dc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:27.515803 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3babee14-98c3-4e23-9e8c-c814c3f17dc7 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:27.516443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3babee14-98c3-4e23-9e8c-c814c3f17dc7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:27.518507 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3babee14-98c3-4e23-9e8c-c814c3f17dc7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:27.550331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 663/1362] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:27 2022] POST /identity/v3/users => generated 285 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:27.555009 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb683a3-d132-4d2d-a217-531e1a7cc8f6 service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:27.583634 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb683a3-d132-4d2d-a217-531e1a7cc8f6 service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:27.592477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deff5c86-24fa-4b2a-9719-49f1d19ada58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:27.593590 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deff5c86-24fa-4b2a-9719-49f1d19ada58 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:27.594046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deff5c86-24fa-4b2a-9719-49f1d19ada58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:27.594645 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deff5c86-24fa-4b2a-9719-49f1d19ada58 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:27.596073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deff5c86-24fa-4b2a-9719-49f1d19ada58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:27.601375 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deff5c86-24fa-4b2a-9719-49f1d19ada58 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:27.625600 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3babee14-98c3-4e23-9e8c-c814c3f17dc7 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:27.635670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 700/1363] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:27 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:27.641721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3babee14-98c3-4e23-9e8c-c814c3f17dc7 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:27.649860 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fbaf7f-eb71-4641-8b33-7dd0013ee515 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:27.651662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fbaf7f-eb71-4641-8b33-7dd0013ee515 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:27.652152 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fbaf7f-eb71-4641-8b33-7dd0013ee515 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:27.652617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fbaf7f-eb71-4641-8b33-7dd0013ee515 admin admin] PATH_INFO: `/v3/projects/692b78de7ff4450ab6d62431ad807c94/users/2db3772fb1ee470598f430a41f952937/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:27.698683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fbaf7f-eb71-4641-8b33-7dd0013ee515 admin admin] RBAC: Authorizing `identity:create_grant(project_id=692b78de7ff4450ab6d62431ad807c94, user_id=2db3772fb1ee470598f430a41f952937, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:27.704805 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fbaf7f-eb71-4641-8b33-7dd0013ee515 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:27.740279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deff5c86-24fa-4b2a-9719-49f1d19ada58 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:27.740804 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 664/1364] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:27 2022] PUT /identity/v3/projects/692b78de7ff4450ab6d62431ad807c94/users/2db3772fb1ee470598f430a41f952937/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:27.752964 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deff5c86-24fa-4b2a-9719-49f1d19ada58 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:27.824392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16091433-b586-4b56-98a8-bfe2823468a6 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:27.824392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16091433-b586-4b56-98a8-bfe2823468a6 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:27.824392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16091433-b586-4b56-98a8-bfe2823468a6 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:27.828057 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 665/1365] 10.222.0.25 () {54 vars in 912 bytes} [Sun Oct 16 00:57:27 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:57:27.843988 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39609fb8-740e-4466-98d1-ac0c47c29e70 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:27.844126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39609fb8-740e-4466-98d1-ac0c47c29e70 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:27.844410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39609fb8-740e-4466-98d1-ac0c47c29e70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:27.958716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcb933d-15a7-4723-8977-e488ba4192fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:27.959446 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcb933d-15a7-4723-8977-e488ba4192fb admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:27.959695 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcb933d-15a7-4723-8977-e488ba4192fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:27.960128 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcb933d-15a7-4723-8977-e488ba4192fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:27.960767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcb933d-15a7-4723-8977-e488ba4192fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:27.970852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcb933d-15a7-4723-8977-e488ba4192fb admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:27.989404 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-39609fb8-740e-4466-98d1-ac0c47c29e70 None None] MFA Rules not processed for user `f4eb1a640186411097cb1e3bb9e92020`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:27.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-39609fb8-740e-4466-98d1-ac0c47c29e70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:28.001454 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 701/1366] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:27 2022] GET /identity/v3/roles => generated 1420 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:28.002507 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcb933d-15a7-4723-8977-e488ba4192fb admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:28.019202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:28.026047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcb933d-15a7-4723-8977-e488ba4192fb admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:28.047099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0314506a-d5e8-40cd-82ad-94b8a19e29da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:28.047810 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0314506a-d5e8-40cd-82ad-94b8a19e29da admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:28.048102 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0314506a-d5e8-40cd-82ad-94b8a19e29da admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:28.048394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0314506a-d5e8-40cd-82ad-94b8a19e29da admin admin] PATH_INFO: `/v3/projects/692b78de7ff4450ab6d62431ad807c94/users/2db3772fb1ee470598f430a41f952937/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:28.051190 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0314506a-d5e8-40cd-82ad-94b8a19e29da admin admin] RBAC: Authorizing `identity:create_grant(project_id=692b78de7ff4450ab6d62431ad807c94, user_id=2db3772fb1ee470598f430a41f952937, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:28.054005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0314506a-d5e8-40cd-82ad-94b8a19e29da admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:28.054652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:28.093424 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 702/1367] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:28 2022] PUT /identity/v3/projects/692b78de7ff4450ab6d62431ad807c94/users/2db3772fb1ee470598f430a41f952937/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:28.095033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d958e0c8-b602-4ff2-b5f5-9030624d3e3f None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:28.095332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d958e0c8-b602-4ff2-b5f5-9030624d3e3f None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:28.095601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d958e0c8-b602-4ff2-b5f5-9030624d3e3f None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:28.097282 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 703/1368] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 16 00:57:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:57:28.098216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55925d7-95f2-4067-ac6a-4746a9cdfa77 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:28.098531 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55925d7-95f2-4067-ac6a-4746a9cdfa77 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:28.098802 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55925d7-95f2-4067-ac6a-4746a9cdfa77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:28.109283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 666/1369] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 16 00:57:27 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:28.133669 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed55ccab-e19e-4ec3-b30c-2537b3086897 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:28.134146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed55ccab-e19e-4ec3-b30c-2537b3086897 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:28.134450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed55ccab-e19e-4ec3-b30c-2537b3086897 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:28.182477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-c55925d7-95f2-4067-ac6a-4746a9cdfa77 None None] MFA Rules not processed for user `2db3772fb1ee470598f430a41f952937`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:28.183857 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c55925d7-95f2-4067-ac6a-4746a9cdfa77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:28.191715 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ed55ccab-e19e-4ec3-b30c-2537b3086897 None None] MFA Rules not processed for user `87785ccd37a642e08a0ad34a4fb1c754`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:28.200485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ed55ccab-e19e-4ec3-b30c-2537b3086897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:28.387018 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 704/1370] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:28 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:28.388033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55925d7-95f2-4067-ac6a-4746a9cdfa77 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:28.391951 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c55925d7-95f2-4067-ac6a-4746a9cdfa77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:28.410357 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 667/1371] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 16 00:57:28 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:57:28.411335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed55ccab-e19e-4ec3-b30c-2537b3086897 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:28.415439 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ed55ccab-e19e-4ec3-b30c-2537b3086897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:28.559202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55925d7-95f2-4067-ac6a-4746a9cdfa77 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:28.572380 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca31518c-2fc8-48ae-876b-6f8a64d046dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:28.573572 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca31518c-2fc8-48ae-876b-6f8a64d046dc service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:28.576754 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca31518c-2fc8-48ae-876b-6f8a64d046dc service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:28.578813 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca31518c-2fc8-48ae-876b-6f8a64d046dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:28.581837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed55ccab-e19e-4ec3-b30c-2537b3086897 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:28.593897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b9ea13-6b54-44d7-80bd-de123ec625f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:28.594663 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b9ea13-6b54-44d7-80bd-de123ec625f6 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:28.595234 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b9ea13-6b54-44d7-80bd-de123ec625f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:28.595610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b9ea13-6b54-44d7-80bd-de123ec625f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:28.603205 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca31518c-2fc8-48ae-876b-6f8a64d046dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:28.613982 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca31518c-2fc8-48ae-876b-6f8a64d046dc service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:28.631392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b9ea13-6b54-44d7-80bd-de123ec625f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:28.635421 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b9ea13-6b54-44d7-80bd-de123ec625f6 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:28.638791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 705/1372] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:28 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:28.661533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 668/1373] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:28 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:28.865336 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca31518c-2fc8-48ae-876b-6f8a64d046dc service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:28.877468 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca31518c-2fc8-48ae-876b-6f8a64d046dc service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:28.897932 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300d12b2-fff4-4072-9150-6350e9b93523 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:28.899862 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300d12b2-fff4-4072-9150-6350e9b93523 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:28.900167 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300d12b2-fff4-4072-9150-6350e9b93523 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:28.900669 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300d12b2-fff4-4072-9150-6350e9b93523 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:28.943251 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b9ea13-6b54-44d7-80bd-de123ec625f6 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:28.946476 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a7b9ea13-6b54-44d7-80bd-de123ec625f6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:28.975044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b9ea13-6b54-44d7-80bd-de123ec625f6 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:28.993147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcecae3c-336b-43d6-adb8-ce138d397b98 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87785ccd37a642e08a0ad34a4fb1c754', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:28.993947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcecae3c-336b-43d6-adb8-ce138d397b98 service placement] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:28.994269 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcecae3c-336b-43d6-adb8-ce138d397b98 service placement] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:28.994647 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcecae3c-336b-43d6-adb8-ce138d397b98 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:29.024863 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcecae3c-336b-43d6-adb8-ce138d397b98 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:29.026541 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcecae3c-336b-43d6-adb8-ce138d397b98 service placement] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:29.053673 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 669/1374] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:28 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:57:29.054606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300d12b2-fff4-4072-9150-6350e9b93523 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:29.056125 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300d12b2-fff4-4072-9150-6350e9b93523 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:29.071988 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 706/1375] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:28 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:30.413910 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcecae3c-336b-43d6-adb8-ce138d397b98 service placement] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:30.416279 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bcecae3c-336b-43d6-adb8-ce138d397b98 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:30.677518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcecae3c-336b-43d6-adb8-ce138d397b98 service placement] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:30.727109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0affcf8-39a8-4304-b61c-6df63624de68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:30.727109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0affcf8-39a8-4304-b61c-6df63624de68 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:30.731105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0affcf8-39a8-4304-b61c-6df63624de68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:30.735160 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0affcf8-39a8-4304-b61c-6df63624de68 admin admin] PATH_INFO: `/v3/users/4c3248ed5595438b924cea73f22598ad` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:30.759363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0affcf8-39a8-4304-b61c-6df63624de68 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4c3248ed5595438b924cea73f22598ad)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:30.777040 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0affcf8-39a8-4304-b61c-6df63624de68 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:30.861464 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:31.054748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300d12b2-fff4-4072-9150-6350e9b93523 service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:31.067100 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c0affcf8-39a8-4304-b61c-6df63624de68 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4c3248ed5595438b924cea73f22598ad'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:31.068233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c0affcf8-39a8-4304-b61c-6df63624de68 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4c3248ed5595438b924cea73f22598ad'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:31.074601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300d12b2-fff4-4072-9150-6350e9b93523 service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:31.096265 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c0affcf8-39a8-4304-b61c-6df63624de68 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4c3248ed5595438b924cea73f22598ad'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:31.098017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c0affcf8-39a8-4304-b61c-6df63624de68 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4c3248ed5595438b924cea73f22598ad'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:31.108699 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68f3134-ecb3-403f-9c81-fb8eac260c20 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87785ccd37a642e08a0ad34a4fb1c754', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:31.111103 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68f3134-ecb3-403f-9c81-fb8eac260c20 service placement] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:31.113944 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68f3134-ecb3-403f-9c81-fb8eac260c20 service placement] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:31.119241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68f3134-ecb3-403f-9c81-fb8eac260c20 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:31.124997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-f68f3134-ecb3-403f-9c81-fb8eac260c20 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:31.156832 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c0affcf8-39a8-4304-b61c-6df63624de68 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4c3248ed5595438b924cea73f22598ad'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:31.170618 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68f3134-ecb3-403f-9c81-fb8eac260c20 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:31.180896 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68f3134-ecb3-403f-9c81-fb8eac260c20 service placement] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:31.212414 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 670/1376] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:57:30 2022] DELETE /identity/v3/users/4c3248ed5595438b924cea73f22598ad => generated 0 bytes in 799 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:31.217554 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bb9cb9-9973-4bcc-a77d-0121b66ebcb2 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:31.217876 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bb9cb9-9973-4bcc-a77d-0121b66ebcb2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:31.218102 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bb9cb9-9973-4bcc-a77d-0121b66ebcb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:31.236505 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 707/1377] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:31 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:57:31.273965 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-59bb9cb9-9973-4bcc-a77d-0121b66ebcb2 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:31.275669 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-59bb9cb9-9973-4bcc-a77d-0121b66ebcb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:31.471523 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 671/1378] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:57:31.591018 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68f3134-ecb3-403f-9c81-fb8eac260c20 service placement] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:31.593520 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-f68f3134-ecb3-403f-9c81-fb8eac260c20 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:31.604478 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:31.613650 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68f3134-ecb3-403f-9c81-fb8eac260c20 service placement] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:31.772424 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a507e6-5fd7-497c-9c8a-04cf0f001c08 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:31.773056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a507e6-5fd7-497c-9c8a-04cf0f001c08 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:31.773340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a507e6-5fd7-497c-9c8a-04cf0f001c08 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:31.773612 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a507e6-5fd7-497c-9c8a-04cf0f001c08 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:31.797199 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a507e6-5fd7-497c-9c8a-04cf0f001c08 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:31.799029 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a507e6-5fd7-497c-9c8a-04cf0f001c08 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:31.825465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 708/1379] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:32.077321 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e061fc05-a1ea-44f4-90ed-56a8b30d2b01 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:32.077703 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e061fc05-a1ea-44f4-90ed-56a8b30d2b01 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:32.077982 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e061fc05-a1ea-44f4-90ed-56a8b30d2b01 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:32.079483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 672/1380] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 16 00:57:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:57:32.100406 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e0d094-4695-48f6-8e64-7fe7d70dca38 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:32.101191 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e0d094-4695-48f6-8e64-7fe7d70dca38 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:32.101191 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e0d094-4695-48f6-8e64-7fe7d70dca38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:32.150293 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-22e0d094-4695-48f6-8e64-7fe7d70dca38 None None] MFA Rules not processed for user `87785ccd37a642e08a0ad34a4fb1c754`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:32.151578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-22e0d094-4695-48f6-8e64-7fe7d70dca38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:32.299134 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e061fc05-a1ea-44f4-90ed-56a8b30d2b01 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:32.300925 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e061fc05-a1ea-44f4-90ed-56a8b30d2b01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:32.357584 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e061fc05-a1ea-44f4-90ed-56a8b30d2b01 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:32.385639 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8093da63-a7e2-4524-83e8-be4cd63bbcdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:32.386360 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8093da63-a7e2-4524-83e8-be4cd63bbcdc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:32.386643 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8093da63-a7e2-4524-83e8-be4cd63bbcdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:32.386932 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8093da63-a7e2-4524-83e8-be4cd63bbcdc admin admin] PATH_INFO: `/v3/users/2a14964894f1476a9c7125f87711a754` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:32.388143 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8093da63-a7e2-4524-83e8-be4cd63bbcdc admin admin] RBAC: Authorizing `identity:delete_user(user_id=2a14964894f1476a9c7125f87711a754)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:32.390813 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8093da63-a7e2-4524-83e8-be4cd63bbcdc admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:32.407460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 709/1381] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 16 00:57:32 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:32.455677 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4e041f-3168-4c21-932f-e6ac8baa5c14 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:32.456352 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4e041f-3168-4c21-932f-e6ac8baa5c14 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:32.458079 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4e041f-3168-4c21-932f-e6ac8baa5c14 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:32.460590 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 710/1382] 10.222.0.25 () {54 vars in 961 bytes} [Sun Oct 16 00:57:32 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 16 00:57:32.469987 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939b35cc-676c-4b46-8dba-a235fcd63eee None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:32.470829 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939b35cc-676c-4b46-8dba-a235fcd63eee None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:32.471946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939b35cc-676c-4b46-8dba-a235fcd63eee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:32.544248 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-939b35cc-676c-4b46-8dba-a235fcd63eee None None] MFA Rules not processed for user `87785ccd37a642e08a0ad34a4fb1c754`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:32.546892 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-939b35cc-676c-4b46-8dba-a235fcd63eee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:32.557678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8093da63-a7e2-4524-83e8-be4cd63bbcdc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2a14964894f1476a9c7125f87711a754'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:32.558652 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 711/1383] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Oct 16 00:57:32 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:32.559304 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8093da63-a7e2-4524-83e8-be4cd63bbcdc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2a14964894f1476a9c7125f87711a754'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:32.568919 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939b35cc-676c-4b46-8dba-a235fcd63eee None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:32.579995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8093da63-a7e2-4524-83e8-be4cd63bbcdc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2a14964894f1476a9c7125f87711a754'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:32.582167 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8093da63-a7e2-4524-83e8-be4cd63bbcdc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2a14964894f1476a9c7125f87711a754'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:32.595197 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939b35cc-676c-4b46-8dba-a235fcd63eee None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:32.610195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8093da63-a7e2-4524-83e8-be4cd63bbcdc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2a14964894f1476a9c7125f87711a754'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:32.615750 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3929e4-30aa-4b6b-9682-f91bbf9c4d6d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87785ccd37a642e08a0ad34a4fb1c754', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:32.616838 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3929e4-30aa-4b6b-9682-f91bbf9c4d6d service placement] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:32.617451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3929e4-30aa-4b6b-9682-f91bbf9c4d6d service placement] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:32.617923 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3929e4-30aa-4b6b-9682-f91bbf9c4d6d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:32.623685 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6d3929e4-30aa-4b6b-9682-f91bbf9c4d6d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:32.711356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3929e4-30aa-4b6b-9682-f91bbf9c4d6d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:32.728545 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3929e4-30aa-4b6b-9682-f91bbf9c4d6d service placement] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:32.729804 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 673/1384] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:57:32 2022] DELETE /identity/v3/users/2a14964894f1476a9c7125f87711a754 => generated 0 bytes in 432 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:57:32.787119 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 712/1385] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:32 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:57:33.673091 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8093da63-a7e2-4524-83e8-be4cd63bbcdc admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:33.674624 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8093da63-a7e2-4524-83e8-be4cd63bbcdc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:33.682775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:33.848905 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8093da63-a7e2-4524-83e8-be4cd63bbcdc admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:33.866322 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f02e4e-62f8-40b4-ad41-39db1bae53b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:33.867499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f02e4e-62f8-40b4-ad41-39db1bae53b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:33.868202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f02e4e-62f8-40b4-ad41-39db1bae53b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:33.868935 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f02e4e-62f8-40b4-ad41-39db1bae53b7 admin admin] PATH_INFO: `/v3/projects/ce82a4cafaac4ccb82a0e07c3212584f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:33.870843 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f02e4e-62f8-40b4-ad41-39db1bae53b7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ce82a4cafaac4ccb82a0e07c3212584f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:33.875975 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f02e4e-62f8-40b4-ad41-39db1bae53b7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:34.058148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-88f02e4e-62f8-40b4-ad41-39db1bae53b7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ce82a4cafaac4ccb82a0e07c3212584f', 'request_id': 'req-88f02e4e-62f8-40b4-ad41-39db1bae53b7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:34.076166 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-88f02e4e-62f8-40b4-ad41-39db1bae53b7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ce82a4cafaac4ccb82a0e07c3212584f', 'request_id': 'req-88f02e4e-62f8-40b4-ad41-39db1bae53b7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:34.086720 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-88f02e4e-62f8-40b4-ad41-39db1bae53b7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ce82a4cafaac4ccb82a0e07c3212584f', 'request_id': 'req-88f02e4e-62f8-40b4-ad41-39db1bae53b7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:34.135197 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-88f02e4e-62f8-40b4-ad41-39db1bae53b7 admin admin] The token cache is being invalidate because project ce82a4cafaac4ccb82a0e07c3212584f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:57:34.137058 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-88f02e4e-62f8-40b4-ad41-39db1bae53b7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:34.140045 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-88f02e4e-62f8-40b4-ad41-39db1bae53b7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:34.148642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 674/1386] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:57:33 2022] DELETE /identity/v3/projects/ce82a4cafaac4ccb82a0e07c3212584f => generated 0 bytes in 477 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:34.152280 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3929e4-30aa-4b6b-9682-f91bbf9c4d6d service placement] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:34.170979 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3929e4-30aa-4b6b-9682-f91bbf9c4d6d service placement] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:34.366816 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037e175f-988f-49aa-af9a-f4503171ce52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:34.367887 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e175f-988f-49aa-af9a-f4503171ce52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:34.368316 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e175f-988f-49aa-af9a-f4503171ce52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:34.368825 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e175f-988f-49aa-af9a-f4503171ce52 admin admin] PATH_INFO: `/v3/users/c421a4762f55475786abf7688aa55d6a` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:34.371697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037e175f-988f-49aa-af9a-f4503171ce52 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c421a4762f55475786abf7688aa55d6a)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:34.374661 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037e175f-988f-49aa-af9a-f4503171ce52 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:34.439711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:34.507461 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-037e175f-988f-49aa-af9a-f4503171ce52 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c421a4762f55475786abf7688aa55d6a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:34.509968 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-037e175f-988f-49aa-af9a-f4503171ce52 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c421a4762f55475786abf7688aa55d6a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:34.525731 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-037e175f-988f-49aa-af9a-f4503171ce52 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c421a4762f55475786abf7688aa55d6a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:34.573919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f02e4e-62f8-40b4-ad41-39db1bae53b7 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:34.576892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-88f02e4e-62f8-40b4-ad41-39db1bae53b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:34.594249 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-037e175f-988f-49aa-af9a-f4503171ce52 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c421a4762f55475786abf7688aa55d6a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:34.609045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-037e175f-988f-49aa-af9a-f4503171ce52 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c421a4762f55475786abf7688aa55d6a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:34.616578 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f02e4e-62f8-40b4-ad41-39db1bae53b7 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:34.624292 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 713/1387] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:57:34 2022] DELETE /identity/v3/users/c421a4762f55475786abf7688aa55d6a => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:34.746446 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dd5e80-b883-4583-9fd4-ac219511f72b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:34.754978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dd5e80-b883-4583-9fd4-ac219511f72b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:34.755427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dd5e80-b883-4583-9fd4-ac219511f72b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:34.773831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:34.780399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:34.780749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:34.782199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc admin admin] PATH_INFO: `/v3/projects/0ebf28d90fe7429d94e65b524ce8d9de` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:34.789189 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ebf28d90fe7429d94e65b524ce8d9de)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:34.791711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:34.869338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-65dd5e80-b883-4583-9fd4-ac219511f72b None None] MFA Rules not processed for user `b01ce2d27bdb4c91a3f55c891f949e00`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:34.872504 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-65dd5e80-b883-4583-9fd4-ac219511f72b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:34.882836 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:34.913123 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ebf28d90fe7429d94e65b524ce8d9de', 'request_id': 'req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:34.931772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ebf28d90fe7429d94e65b524ce8d9de', 'request_id': 'req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:34.953140 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ebf28d90fe7429d94e65b524ce8d9de', 'request_id': 'req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:34.984117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc admin admin] The token cache is being invalidate because project 0ebf28d90fe7429d94e65b524ce8d9de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:57:34.984429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:34.987283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:34.994200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 675/1388] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:57:34 2022] DELETE /identity/v3/projects/0ebf28d90fe7429d94e65b524ce8d9de => generated 0 bytes in 418 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:34.998628 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:35.000534 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:35.044006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 714/1389] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:34 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 290 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:35.045975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9527d3-d690-4c2d-9289-d0d86c650f0b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:35.046273 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9527d3-d690-4c2d-9289-d0d86c650f0b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:35.046558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9527d3-d690-4c2d-9289-d0d86c650f0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:35.089380 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-bd9527d3-d690-4c2d-9289-d0d86c650f0b None None] MFA Rules not processed for user `b01ce2d27bdb4c91a3f55c891f949e00`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:35.091181 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-bd9527d3-d690-4c2d-9289-d0d86c650f0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:35.113509 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 715/1390] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:35 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:57:35.121536 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9527d3-d690-4c2d-9289-d0d86c650f0b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:35.124407 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-bd9527d3-d690-4c2d-9289-d0d86c650f0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:35.314976 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae96416-1c0c-4f0c-9129-0a3c36a389fc admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:35.348794 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a49e3eb-f96d-48da-9783-2e734be0f4ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:35.349782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a49e3eb-f96d-48da-9783-2e734be0f4ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:35.356460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a49e3eb-f96d-48da-9783-2e734be0f4ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:35.357255 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a49e3eb-f96d-48da-9783-2e734be0f4ce admin admin] PATH_INFO: `/v3/users/e5773c56e4954981be04d587c61be9d6` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:35.363187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a49e3eb-f96d-48da-9783-2e734be0f4ce admin admin] RBAC: Authorizing `identity:delete_user(user_id=e5773c56e4954981be04d587c61be9d6)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:35.364639 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9527d3-d690-4c2d-9289-d0d86c650f0b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:35.377552 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a49e3eb-f96d-48da-9783-2e734be0f4ce admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:35.380268 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0491208-a4fe-4934-9452-f8c2147f80d6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:35.381035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0491208-a4fe-4934-9452-f8c2147f80d6 service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:35.382905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0491208-a4fe-4934-9452-f8c2147f80d6 service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:35.383316 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0491208-a4fe-4934-9452-f8c2147f80d6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:35.417396 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0491208-a4fe-4934-9452-f8c2147f80d6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:35.421878 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0491208-a4fe-4934-9452-f8c2147f80d6 service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:35.467169 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 716/1391] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:35 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:35.475171 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd58520-575e-4f3c-ad3f-23d8e8f5b051 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:35.475706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd58520-575e-4f3c-ad3f-23d8e8f5b051 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:35.477058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd58520-575e-4f3c-ad3f-23d8e8f5b051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:35.546552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:35.584101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-cfd58520-575e-4f3c-ad3f-23d8e8f5b051 None None] MFA Rules not processed for user `466f2f8ddd184a98bdb5e922818905cc`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:35.586489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-cfd58520-575e-4f3c-ad3f-23d8e8f5b051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:35.594879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4a49e3eb-f96d-48da-9783-2e734be0f4ce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e5773c56e4954981be04d587c61be9d6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:35.610420 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4a49e3eb-f96d-48da-9783-2e734be0f4ce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e5773c56e4954981be04d587c61be9d6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:35.636146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4a49e3eb-f96d-48da-9783-2e734be0f4ce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e5773c56e4954981be04d587c61be9d6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:35.636972 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4a49e3eb-f96d-48da-9783-2e734be0f4ce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e5773c56e4954981be04d587c61be9d6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:35.690802 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4a49e3eb-f96d-48da-9783-2e734be0f4ce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e5773c56e4954981be04d587c61be9d6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:35.729549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 676/1392] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:57:34 2022] DELETE /identity/v3/users/e5773c56e4954981be04d587c61be9d6 => generated 0 bytes in 735 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:35.730509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a49e3eb-f96d-48da-9783-2e734be0f4ce admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:35.732138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4a49e3eb-f96d-48da-9783-2e734be0f4ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:35.848177 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 717/1393] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:35 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 381 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:35.873356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd58520-575e-4f3c-ad3f-23d8e8f5b051 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:35.875543 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-cfd58520-575e-4f3c-ad3f-23d8e8f5b051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:35.889365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:35.919783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd58520-575e-4f3c-ad3f-23d8e8f5b051 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:35.940891 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a49e3eb-f96d-48da-9783-2e734be0f4ce admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:35.955082 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f75b4f2-b355-44b9-a756-0d157b250fd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:35.955755 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f75b4f2-b355-44b9-a756-0d157b250fd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:35.956039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f75b4f2-b355-44b9-a756-0d157b250fd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:35.956344 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f75b4f2-b355-44b9-a756-0d157b250fd0 admin admin] PATH_INFO: `/v3/projects/7d6e0d35b8394e6fb979a7ba3544d31f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:35.959819 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f75b4f2-b355-44b9-a756-0d157b250fd0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7d6e0d35b8394e6fb979a7ba3544d31f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:35.961788 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f75b4f2-b355-44b9-a756-0d157b250fd0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:36.084426 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f75b4f2-b355-44b9-a756-0d157b250fd0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7d6e0d35b8394e6fb979a7ba3544d31f', 'request_id': 'req-6f75b4f2-b355-44b9-a756-0d157b250fd0'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:36.101619 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f75b4f2-b355-44b9-a756-0d157b250fd0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7d6e0d35b8394e6fb979a7ba3544d31f', 'request_id': 'req-6f75b4f2-b355-44b9-a756-0d157b250fd0'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:36.110041 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f75b4f2-b355-44b9-a756-0d157b250fd0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7d6e0d35b8394e6fb979a7ba3544d31f', 'request_id': 'req-6f75b4f2-b355-44b9-a756-0d157b250fd0'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:36.165868 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f75b4f2-b355-44b9-a756-0d157b250fd0 admin admin] The token cache is being invalidate because project 7d6e0d35b8394e6fb979a7ba3544d31f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:57:36.167030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79971711-30eb-4b2f-9571-78b0e1002a1b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:36.168785 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f75b4f2-b355-44b9-a756-0d157b250fd0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:36.170472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79971711-30eb-4b2f-9571-78b0e1002a1b service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:36.170766 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79971711-30eb-4b2f-9571-78b0e1002a1b service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:36.171075 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79971711-30eb-4b2f-9571-78b0e1002a1b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:36.172239 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f75b4f2-b355-44b9-a756-0d157b250fd0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:36.176568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 677/1394] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:57:35 2022] DELETE /identity/v3/projects/7d6e0d35b8394e6fb979a7ba3544d31f => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:36.182168 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-79971711-30eb-4b2f-9571-78b0e1002a1b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:36.182505 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d317ef-18bb-4a7d-b1dc-8a6788a2b42f None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:36.182797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d317ef-18bb-4a7d-b1dc-8a6788a2b42f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:36.183107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d317ef-18bb-4a7d-b1dc-8a6788a2b42f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:36.220608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d8d317ef-18bb-4a7d-b1dc-8a6788a2b42f None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:36.223696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d8d317ef-18bb-4a7d-b1dc-8a6788a2b42f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:36.460471 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 678/1395] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:36.466594 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95634f67-a993-4595-8fa9-3d2c10a9404c None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:36.467128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95634f67-a993-4595-8fa9-3d2c10a9404c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:36.467416 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95634f67-a993-4595-8fa9-3d2c10a9404c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:36.500329 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79971711-30eb-4b2f-9571-78b0e1002a1b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:36.503728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79971711-30eb-4b2f-9571-78b0e1002a1b service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:36.561921 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 718/1396] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:35 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 689 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:57:36.564447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-95634f67-a993-4595-8fa9-3d2c10a9404c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:36.571549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-95634f67-a993-4595-8fa9-3d2c10a9404c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:36.582058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79971711-30eb-4b2f-9571-78b0e1002a1b service nova] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:36.586311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-79971711-30eb-4b2f-9571-78b0e1002a1b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:36.608860 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 679/1397] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:57:36.616384 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95634f67-a993-4595-8fa9-3d2c10a9404c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:36.638727 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79971711-30eb-4b2f-9571-78b0e1002a1b service nova] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:36.655662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95634f67-a993-4595-8fa9-3d2c10a9404c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:36.665725 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bba4cb9-6a91-4fe4-9c46-95fd2774d823 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:36.666396 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bba4cb9-6a91-4fe4-9c46-95fd2774d823 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:36.666682 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bba4cb9-6a91-4fe4-9c46-95fd2774d823 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:36.675209 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bba4cb9-6a91-4fe4-9c46-95fd2774d823 admin admin] PATH_INFO: `/v3/projects/1daba3a4330d42c8a14ccdf105b6738c` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:36.678791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bba4cb9-6a91-4fe4-9c46-95fd2774d823 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1daba3a4330d42c8a14ccdf105b6738c)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:36.680883 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bba4cb9-6a91-4fe4-9c46-95fd2774d823 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:36.686898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7815ed-0c51-4bcb-af61-79c1c5b85df3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:36.686898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7815ed-0c51-4bcb-af61-79c1c5b85df3 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:36.686898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7815ed-0c51-4bcb-af61-79c1c5b85df3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:36.686898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7815ed-0c51-4bcb-af61-79c1c5b85df3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:36.687338 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7815ed-0c51-4bcb-af61-79c1c5b85df3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:57:36.687338 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7815ed-0c51-4bcb-af61-79c1c5b85df3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:36.688634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f7815ed-0c51-4bcb-af61-79c1c5b85df3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:36.744207 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 680/1398] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:57:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:57:36.749563 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7815ed-0c51-4bcb-af61-79c1c5b85df3 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:36.785895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f7815ed-0c51-4bcb-af61-79c1c5b85df3 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:36.805787 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837d3fa6-d33e-4e9c-9d0b-a0f5850fd14f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:36.815107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837d3fa6-d33e-4e9c-9d0b-a0f5850fd14f admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:36.815511 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837d3fa6-d33e-4e9c-9d0b-a0f5850fd14f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:36.815801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837d3fa6-d33e-4e9c-9d0b-a0f5850fd14f admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:36.816664 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837d3fa6-d33e-4e9c-9d0b-a0f5850fd14f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:36.823855 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:36.829779 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837d3fa6-d33e-4e9c-9d0b-a0f5850fd14f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:36.855361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-0bba4cb9-6a91-4fe4-9c46-95fd2774d823 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1daba3a4330d42c8a14ccdf105b6738c', 'request_id': 'req-0bba4cb9-6a91-4fe4-9c46-95fd2774d823'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:36.878883 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-0bba4cb9-6a91-4fe4-9c46-95fd2774d823 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1daba3a4330d42c8a14ccdf105b6738c', 'request_id': 'req-0bba4cb9-6a91-4fe4-9c46-95fd2774d823'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:36.915665 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-0bba4cb9-6a91-4fe4-9c46-95fd2774d823 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1daba3a4330d42c8a14ccdf105b6738c', 'request_id': 'req-0bba4cb9-6a91-4fe4-9c46-95fd2774d823'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:36.961232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-0bba4cb9-6a91-4fe4-9c46-95fd2774d823 admin admin] The token cache is being invalidate because project 1daba3a4330d42c8a14ccdf105b6738c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:57:36.961232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-0bba4cb9-6a91-4fe4-9c46-95fd2774d823 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:36.969457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-0bba4cb9-6a91-4fe4-9c46-95fd2774d823 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:36.970988 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 681/1399] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:36 2022] POST /identity/v3/projects => generated 369 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:36.972447 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 719/1400] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:57:36 2022] DELETE /identity/v3/projects/1daba3a4330d42c8a14ccdf105b6738c => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:57:36.979845 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837d3fa6-d33e-4e9c-9d0b-a0f5850fd14f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:36.981102 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8faab3-66a8-41c3-9d50-85292afa94db None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:36.982794 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8faab3-66a8-41c3-9d50-85292afa94db None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:36.983891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8faab3-66a8-41c3-9d50-85292afa94db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:36.985894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-837d3fa6-d33e-4e9c-9d0b-a0f5850fd14f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:37.073290 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-eb8faab3-66a8-41c3-9d50-85292afa94db None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:37.087585 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-eb8faab3-66a8-41c3-9d50-85292afa94db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:37.118220 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:37.249284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837d3fa6-d33e-4e9c-9d0b-a0f5850fd14f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:37.296811 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8779da-154f-4984-9c3b-f4559bf506cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:37.297464 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8779da-154f-4984-9c3b-f4559bf506cc admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:37.297743 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8779da-154f-4984-9c3b-f4559bf506cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:37.298019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8779da-154f-4984-9c3b-f4559bf506cc admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:37.298662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf8779da-154f-4984-9c3b-f4559bf506cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:37.300670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf8779da-154f-4984-9c3b-f4559bf506cc admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:37.311890 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 720/1401] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:57:37.313615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0214c14-ae61-4269-b2ae-3dc364468263 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:37.313615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0214c14-ae61-4269-b2ae-3dc364468263 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:37.313964 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0214c14-ae61-4269-b2ae-3dc364468263 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:37.377775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 682/1402] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:36 2022] POST /identity/v3/users => generated 285 bytes in 403 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:37.377775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdeb371-64b0-4a36-8ed1-9140b1056fa0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:37.377775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdeb371-64b0-4a36-8ed1-9140b1056fa0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:37.377775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdeb371-64b0-4a36-8ed1-9140b1056fa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:37.428665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-7fdeb371-64b0-4a36-8ed1-9140b1056fa0 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:37.428665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7fdeb371-64b0-4a36-8ed1-9140b1056fa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:37.457355 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-b0214c14-ae61-4269-b2ae-3dc364468263 None None] MFA Rules not processed for user `33b136afbf8a4d98a9d62053b68707e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:37.458126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 683/1403] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:37.459870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b0214c14-ae61-4269-b2ae-3dc364468263 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:37.463496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdeb371-64b0-4a36-8ed1-9140b1056fa0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:37.494596 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdeb371-64b0-4a36-8ed1-9140b1056fa0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:37.515278 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c2aa6a-9f9a-492c-a207-0d0ec01af81c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:37.517830 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c2aa6a-9f9a-492c-a207-0d0ec01af81c admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:37.518570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c2aa6a-9f9a-492c-a207-0d0ec01af81c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:37.518570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c2aa6a-9f9a-492c-a207-0d0ec01af81c admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:37.520146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c2aa6a-9f9a-492c-a207-0d0ec01af81c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:37.522343 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c2aa6a-9f9a-492c-a207-0d0ec01af81c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:37.542958 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 684/1404] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:37 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:37.546083 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c2aa6a-9f9a-492c-a207-0d0ec01af81c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:37.586331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c2aa6a-9f9a-492c-a207-0d0ec01af81c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:37.623627 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b938cf8-8e8f-428c-bbe8-7f3279686bca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:37.624489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b938cf8-8e8f-428c-bbe8-7f3279686bca admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:37.627870 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b938cf8-8e8f-428c-bbe8-7f3279686bca admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:37.628258 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b938cf8-8e8f-428c-bbe8-7f3279686bca admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:37.630752 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b938cf8-8e8f-428c-bbe8-7f3279686bca admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:57:37.633995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b938cf8-8e8f-428c-bbe8-7f3279686bca admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:37.641455 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b938cf8-8e8f-428c-bbe8-7f3279686bca admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:37.656844 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 721/1405] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 16 00:57:37 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 344 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:37.676582 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0214c14-ae61-4269-b2ae-3dc364468263 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:37.695928 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 685/1406] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:57:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:57:37.696956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b938cf8-8e8f-428c-bbe8-7f3279686bca admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:37.730959 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0214c14-ae61-4269-b2ae-3dc364468263 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:37.748850 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142458d7-c510-4652-a8c6-bc722e2c14d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:37.748850 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142458d7-c510-4652-a8c6-bc722e2c14d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:37.749136 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b938cf8-8e8f-428c-bbe8-7f3279686bca admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:37.764319 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142458d7-c510-4652-a8c6-bc722e2c14d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:37.766444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142458d7-c510-4652-a8c6-bc722e2c14d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:37.767608 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142458d7-c510-4652-a8c6-bc722e2c14d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:37.786465 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ec9192-339c-4667-9287-a8a6d9e67f2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:37.787156 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ec9192-339c-4667-9287-a8a6d9e67f2d admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:37.787445 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ec9192-339c-4667-9287-a8a6d9e67f2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:37.787720 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ec9192-339c-4667-9287-a8a6d9e67f2d admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:37.790544 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ec9192-339c-4667-9287-a8a6d9e67f2d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:37.791513 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142458d7-c510-4652-a8c6-bc722e2c14d5 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:37.800852 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ec9192-339c-4667-9287-a8a6d9e67f2d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:37.832665 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 722/1407] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:37 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:37.834986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142458d7-c510-4652-a8c6-bc722e2c14d5 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:37.864262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142458d7-c510-4652-a8c6-bc722e2c14d5 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:37.883436 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a5f9b4-edd5-44cb-b83a-a2d9e1f9b016 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:37.884315 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a5f9b4-edd5-44cb-b83a-a2d9e1f9b016 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:37.889012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 686/1408] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:37 2022] POST /identity/v3/projects => generated 367 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:37.889937 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a5f9b4-edd5-44cb-b83a-a2d9e1f9b016 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:37.891680 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a5f9b4-edd5-44cb-b83a-a2d9e1f9b016 admin admin] PATH_INFO: `/v3/projects/1e231fb307c24d1897d19a612aaa1259/users/8a516b2896424ca7a3169d667ab3b8b3/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:37.904818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ec9192-339c-4667-9287-a8a6d9e67f2d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:37.954909 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a5f9b4-edd5-44cb-b83a-a2d9e1f9b016 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e231fb307c24d1897d19a612aaa1259, user_id=8a516b2896424ca7a3169d667ab3b8b3, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:37.959983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a5f9b4-edd5-44cb-b83a-a2d9e1f9b016 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:37.982325 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:37.995602 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 723/1409] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:37 2022] PUT /identity/v3/projects/1e231fb307c24d1897d19a612aaa1259/users/8a516b2896424ca7a3169d667ab3b8b3/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:37.998412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04902e31-76e4-42e6-ab36-a0e30fba231f None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:38.001519 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04902e31-76e4-42e6-ab36-a0e30fba231f None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:38.003139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04902e31-76e4-42e6-ab36-a0e30fba231f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:38.103086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-04902e31-76e4-42e6-ab36-a0e30fba231f None None] MFA Rules not processed for user `ccece85a85fb48c4887b52c96c5bc642`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:38.104822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-04902e31-76e4-42e6-ab36-a0e30fba231f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:38.111748 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ec9192-339c-4667-9287-a8a6d9e67f2d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:38.325217 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 724/1410] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:37 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 328 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:38.326911 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04902e31-76e4-42e6-ab36-a0e30fba231f None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:38.332033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-04902e31-76e4-42e6-ab36-a0e30fba231f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:38.404152 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3157cdd-9587-4ac1-a820-af2791aa0dee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:38.413277 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3157cdd-9587-4ac1-a820-af2791aa0dee admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:38.413587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3157cdd-9587-4ac1-a820-af2791aa0dee admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:38.413865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3157cdd-9587-4ac1-a820-af2791aa0dee admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:38.423305 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3157cdd-9587-4ac1-a820-af2791aa0dee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:38.430667 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3157cdd-9587-4ac1-a820-af2791aa0dee admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:38.504459 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 687/1411] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:37 2022] POST /identity/v3/users => generated 284 bytes in 609 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:38.506374 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66ccd5f-7ef8-4ef7-8741-ce9ca71374a2 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:38.507672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66ccd5f-7ef8-4ef7-8741-ce9ca71374a2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:38.507986 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66ccd5f-7ef8-4ef7-8741-ce9ca71374a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:38.541481 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:38.602484 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e66ccd5f-7ef8-4ef7-8741-ce9ca71374a2 None None] MFA Rules not processed for user `8a516b2896424ca7a3169d667ab3b8b3`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:38.604037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e66ccd5f-7ef8-4ef7-8741-ce9ca71374a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:38.605243 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04902e31-76e4-42e6-ab36-a0e30fba231f None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:38.619605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422fae1e-01b1-4fab-a4e9-dbdf364dcf4b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de64d498da78448394b0d4d1851a2789', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:38.620712 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422fae1e-01b1-4fab-a4e9-dbdf364dcf4b service swift] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:38.622260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422fae1e-01b1-4fab-a4e9-dbdf364dcf4b service swift] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:38.625516 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422fae1e-01b1-4fab-a4e9-dbdf364dcf4b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:38.788833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422fae1e-01b1-4fab-a4e9-dbdf364dcf4b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:38.790448 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422fae1e-01b1-4fab-a4e9-dbdf364dcf4b service swift] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:38.815778 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 725/1412] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 16 00:57:38 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 490 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 16 00:57:38.822154 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 688/1413] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:38 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 317 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:38.835364 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422fae1e-01b1-4fab-a4e9-dbdf364dcf4b service swift] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:38.842095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-422fae1e-01b1-4fab-a4e9-dbdf364dcf4b service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:38.851881 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66ccd5f-7ef8-4ef7-8741-ce9ca71374a2 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:38.898891 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66ccd5f-7ef8-4ef7-8741-ce9ca71374a2 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:38.917505 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360b067b-992b-4216-8c23-aa7fb6eeee33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:38.923695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360b067b-992b-4216-8c23-aa7fb6eeee33 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:38.924208 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360b067b-992b-4216-8c23-aa7fb6eeee33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:38.934959 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360b067b-992b-4216-8c23-aa7fb6eeee33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:38.935928 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360b067b-992b-4216-8c23-aa7fb6eeee33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:38.940450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360b067b-992b-4216-8c23-aa7fb6eeee33 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:38.946093 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:39.000455 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 689/1414] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:38 2022] GET /identity/v3/roles => generated 1420 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:39.006662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc6a70c-5e43-4e09-acf6-da145c732920 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:39.006662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc6a70c-5e43-4e09-acf6-da145c732920 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:39.006662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc6a70c-5e43-4e09-acf6-da145c732920 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:39.102998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4fc6a70c-5e43-4e09-acf6-da145c732920 None None] MFA Rules not processed for user `b01ce2d27bdb4c91a3f55c891f949e00`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:39.108395 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4fc6a70c-5e43-4e09-acf6-da145c732920 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:39.279085 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422fae1e-01b1-4fab-a4e9-dbdf364dcf4b service swift] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:39.302246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa84b37-08e8-475f-b328-e8d93c04fc9b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:39.304333 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa84b37-08e8-475f-b328-e8d93c04fc9b service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:39.308203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa84b37-08e8-475f-b328-e8d93c04fc9b service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:39.313241 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 690/1415] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:39 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:39.316741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc6a70c-5e43-4e09-acf6-da145c732920 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:39.324392 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa84b37-08e8-475f-b328-e8d93c04fc9b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:39.368206 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc6a70c-5e43-4e09-acf6-da145c732920 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:39.391030 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c45c48-882f-4c86-ad4a-6b808682b650 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:39.392868 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa84b37-08e8-475f-b328-e8d93c04fc9b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:39.393902 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c45c48-882f-4c86-ad4a-6b808682b650 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:39.394434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c45c48-882f-4c86-ad4a-6b808682b650 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:39.397006 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c45c48-882f-4c86-ad4a-6b808682b650 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:39.402728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa84b37-08e8-475f-b328-e8d93c04fc9b service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:39.403405 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c45c48-882f-4c86-ad4a-6b808682b650 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:39.413918 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c45c48-882f-4c86-ad4a-6b808682b650 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:39.486686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 726/1416] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:38 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 652 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:39.491008 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa84b37-08e8-475f-b328-e8d93c04fc9b service nova] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:39.529992 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 691/1417] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:39 2022] POST /identity/v3/projects => generated 369 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:39.533103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c45c48-882f-4c86-ad4a-6b808682b650 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:39.550525 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa84b37-08e8-475f-b328-e8d93c04fc9b service nova] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:39.563475 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-59c45c48-882f-4c86-ad4a-6b808682b650 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:39.854499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c45c48-882f-4c86-ad4a-6b808682b650 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:39.879950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7449cac2-ac3c-4c5c-8ce3-e98eb90a6913 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:39.886600 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7449cac2-ac3c-4c5c-8ce3-e98eb90a6913 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:39.889359 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7449cac2-ac3c-4c5c-8ce3-e98eb90a6913 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:39.891773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7449cac2-ac3c-4c5c-8ce3-e98eb90a6913 admin admin] PATH_INFO: `/v3/users/cae4e8c5ef13480ab1939587a4fe7d6a` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:39.895144 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f5b66b-54a9-4882-a3de-16de654f3c7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:39.899355 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f5b66b-54a9-4882-a3de-16de654f3c7d admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:39.899856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f5b66b-54a9-4882-a3de-16de654f3c7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:39.900234 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f5b66b-54a9-4882-a3de-16de654f3c7d admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:39.901066 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f5b66b-54a9-4882-a3de-16de654f3c7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:39.904934 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f5b66b-54a9-4882-a3de-16de654f3c7d admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:39.924252 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7449cac2-ac3c-4c5c-8ce3-e98eb90a6913 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cae4e8c5ef13480ab1939587a4fe7d6a)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:39.938436 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7449cac2-ac3c-4c5c-8ce3-e98eb90a6913 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:39.967808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 727/1418] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:39 2022] GET /identity/v3/roles => generated 1420 bytes in 477 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:39.978385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:39.993935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f5b66b-54a9-4882-a3de-16de654f3c7d admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:40.036416 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f5b66b-54a9-4882-a3de-16de654f3c7d admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:40.056581 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf5feeb-a167-4eaa-953b-c1f792e2d8bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:40.057602 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf5feeb-a167-4eaa-953b-c1f792e2d8bc admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:40.058101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf5feeb-a167-4eaa-953b-c1f792e2d8bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:40.058589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf5feeb-a167-4eaa-953b-c1f792e2d8bc admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:40.059655 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf5feeb-a167-4eaa-953b-c1f792e2d8bc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:40.065000 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf5feeb-a167-4eaa-953b-c1f792e2d8bc admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:40.111620 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7449cac2-ac3c-4c5c-8ce3-e98eb90a6913 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cae4e8c5ef13480ab1939587a4fe7d6a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:40.114727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7449cac2-ac3c-4c5c-8ce3-e98eb90a6913 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cae4e8c5ef13480ab1939587a4fe7d6a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:40.126581 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 728/1419] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:39 2022] POST /identity/v3/users => generated 291 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:40.127855 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf5feeb-a167-4eaa-953b-c1f792e2d8bc admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:40.141589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7449cac2-ac3c-4c5c-8ce3-e98eb90a6913 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cae4e8c5ef13480ab1939587a4fe7d6a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:40.142479 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7449cac2-ac3c-4c5c-8ce3-e98eb90a6913 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cae4e8c5ef13480ab1939587a4fe7d6a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:40.143030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4bf5feeb-a167-4eaa-953b-c1f792e2d8bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:40.165470 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf5feeb-a167-4eaa-953b-c1f792e2d8bc admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:40.193301 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717a3690-4682-4c99-a88e-93762fa9a125 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:40.197106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717a3690-4682-4c99-a88e-93762fa9a125 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:40.197720 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717a3690-4682-4c99-a88e-93762fa9a125 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:40.198215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717a3690-4682-4c99-a88e-93762fa9a125 admin admin] PATH_INFO: `/v3/projects/ebf01e91c24e4380a3115bf63dda7bbd/users/480e0ce54ba04565999cb976c0b7d814/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:40.203635 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7449cac2-ac3c-4c5c-8ce3-e98eb90a6913 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cae4e8c5ef13480ab1939587a4fe7d6a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:40.243099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 692/1420] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:57:39 2022] DELETE /identity/v3/users/cae4e8c5ef13480ab1939587a4fe7d6a => generated 0 bytes in 716 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:40.246233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7449cac2-ac3c-4c5c-8ce3-e98eb90a6913 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:40.248844 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7449cac2-ac3c-4c5c-8ce3-e98eb90a6913 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:40.282603 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717a3690-4682-4c99-a88e-93762fa9a125 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ebf01e91c24e4380a3115bf63dda7bbd, user_id=480e0ce54ba04565999cb976c0b7d814, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:40.297256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717a3690-4682-4c99-a88e-93762fa9a125 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:40.355189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 729/1421] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:40 2022] PUT /identity/v3/projects/ebf01e91c24e4380a3115bf63dda7bbd/users/480e0ce54ba04565999cb976c0b7d814/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:40.357624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f357619-c99c-4e34-9aa5-2271e143f6d8 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:40.364412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f357619-c99c-4e34-9aa5-2271e143f6d8 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:40.367506 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f357619-c99c-4e34-9aa5-2271e143f6d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:40.474823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-2f357619-c99c-4e34-9aa5-2271e143f6d8 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:40.482389 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-2f357619-c99c-4e34-9aa5-2271e143f6d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:40.605222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7449cac2-ac3c-4c5c-8ce3-e98eb90a6913 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:40.705001 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:40.757615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 730/1422] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 402 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:40.759648 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292ba7c9-feae-4d4c-997f-5c3924d42f72 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:40.760062 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292ba7c9-feae-4d4c-997f-5c3924d42f72 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:40.760543 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292ba7c9-feae-4d4c-997f-5c3924d42f72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:40.890556 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-292ba7c9-feae-4d4c-997f-5c3924d42f72 None None] MFA Rules not processed for user `480e0ce54ba04565999cb976c0b7d814`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:40.900116 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-292ba7c9-feae-4d4c-997f-5c3924d42f72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:40.972681 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcbd47a-9c3b-4aa2-bf36-64a97429f0ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:40.979117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcbd47a-9c3b-4aa2-bf36-64a97429f0ee admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:40.984250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcbd47a-9c3b-4aa2-bf36-64a97429f0ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:40.987448 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcbd47a-9c3b-4aa2-bf36-64a97429f0ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:40.988187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcbd47a-9c3b-4aa2-bf36-64a97429f0ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:41.002501 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcbd47a-9c3b-4aa2-bf36-64a97429f0ee admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:41.042935 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 693/1423] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:40 2022] GET /identity/v3/roles => generated 1420 bytes in 799 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:41.047544 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcbd47a-9c3b-4aa2-bf36-64a97429f0ee admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:41.051971 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0bcbd47a-9c3b-4aa2-bf36-64a97429f0ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:41.295772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:41.296679 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 731/1424] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:40 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 539 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:41.297951 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292ba7c9-feae-4d4c-997f-5c3924d42f72 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:41.333619 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292ba7c9-feae-4d4c-997f-5c3924d42f72 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:41.346488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06885e3-d028-4821-aea8-5a0adc7e3e34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:41.347241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06885e3-d028-4821-aea8-5a0adc7e3e34 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:41.347530 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06885e3-d028-4821-aea8-5a0adc7e3e34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:41.347811 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06885e3-d028-4821-aea8-5a0adc7e3e34 admin admin] PATH_INFO: `/v3/projects/7d45f968521b40bf8807ee060da82f1e/users/11b70bf7d4654cc9b5a92d11eaf659df/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:41.398687 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06885e3-d028-4821-aea8-5a0adc7e3e34 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7d45f968521b40bf8807ee060da82f1e, user_id=11b70bf7d4654cc9b5a92d11eaf659df, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:41.401667 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06885e3-d028-4821-aea8-5a0adc7e3e34 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:41.422105 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 732/1425] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:41 2022] PUT /identity/v3/projects/7d45f968521b40bf8807ee060da82f1e/users/11b70bf7d4654cc9b5a92d11eaf659df/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:41.424092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1947d476-77b9-479f-b0dc-fb89e1a009e4 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:41.424387 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1947d476-77b9-479f-b0dc-fb89e1a009e4 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:41.424702 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1947d476-77b9-479f-b0dc-fb89e1a009e4 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:41.426559 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 733/1426] 10.222.0.25 () {54 vars in 912 bytes} [Sun Oct 16 00:57:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:57:41.427797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1947d476-77b9-479f-b0dc-fb89e1a009e4 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:41.457945 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcbd47a-9c3b-4aa2-bf36-64a97429f0ee admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:41.580732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1947d476-77b9-479f-b0dc-fb89e1a009e4 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:41.589658 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9420c6e9-1bf8-4380-8d73-d0a6cd2c2ea4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:41.596500 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9420c6e9-1bf8-4380-8d73-d0a6cd2c2ea4 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:41.597184 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9420c6e9-1bf8-4380-8d73-d0a6cd2c2ea4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:41.597991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9420c6e9-1bf8-4380-8d73-d0a6cd2c2ea4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:41.599712 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9420c6e9-1bf8-4380-8d73-d0a6cd2c2ea4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:41.605340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9420c6e9-1bf8-4380-8d73-d0a6cd2c2ea4 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:41.674965 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9db636-e87e-4cd8-a00c-614edf0cee5e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:41.675626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9db636-e87e-4cd8-a00c-614edf0cee5e service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:41.682042 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9db636-e87e-4cd8-a00c-614edf0cee5e service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:41.684262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 734/1427] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:41 2022] POST /identity/v3/projects => generated 369 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:41.685553 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9db636-e87e-4cd8-a00c-614edf0cee5e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:41.686898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9420c6e9-1bf8-4380-8d73-d0a6cd2c2ea4 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:41.741075 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9db636-e87e-4cd8-a00c-614edf0cee5e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:41.748788 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9db636-e87e-4cd8-a00c-614edf0cee5e service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:41.808485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 694/1428] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 765 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:41.818341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768f1f95-cc51-432e-a6dc-9ab50d35fcbd None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:41.818714 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768f1f95-cc51-432e-a6dc-9ab50d35fcbd None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:41.818947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768f1f95-cc51-432e-a6dc-9ab50d35fcbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:41.878445 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:41.907423 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-768f1f95-cc51-432e-a6dc-9ab50d35fcbd None None] MFA Rules not processed for user `f4eb1a640186411097cb1e3bb9e92020`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:41.910393 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-768f1f95-cc51-432e-a6dc-9ab50d35fcbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:41.976174 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9420c6e9-1bf8-4380-8d73-d0a6cd2c2ea4 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:41.992945 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75946228-8d2f-486d-b070-8d797b8c6905 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:41.995110 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75946228-8d2f-486d-b070-8d797b8c6905 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:41.997104 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75946228-8d2f-486d-b070-8d797b8c6905 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:41.997977 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75946228-8d2f-486d-b070-8d797b8c6905 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:41.999486 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75946228-8d2f-486d-b070-8d797b8c6905 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:42.000412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75946228-8d2f-486d-b070-8d797b8c6905 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:42.074607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 735/1429] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:41 2022] GET /identity/v3/roles => generated 1420 bytes in 389 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:42.082266 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75946228-8d2f-486d-b070-8d797b8c6905 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:42.111804 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75946228-8d2f-486d-b070-8d797b8c6905 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:42.123254 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 695/1430] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 16 00:57:41 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 314 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:42.124625 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768f1f95-cc51-432e-a6dc-9ab50d35fcbd None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:42.129237 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3d05f8-1c95-48f8-b4da-f14d424d2c98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:42.133417 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3d05f8-1c95-48f8-b4da-f14d424d2c98 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:42.133742 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3d05f8-1c95-48f8-b4da-f14d424d2c98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:42.134030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3d05f8-1c95-48f8-b4da-f14d424d2c98 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:42.140595 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3d05f8-1c95-48f8-b4da-f14d424d2c98 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:42.147649 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3d05f8-1c95-48f8-b4da-f14d424d2c98 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:42.151976 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768f1f95-cc51-432e-a6dc-9ab50d35fcbd None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:42.173682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6baeea9a-ed64-4bf6-b9f8-ccf2351e07d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:42.174775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6baeea9a-ed64-4bf6-b9f8-ccf2351e07d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:42.175258 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6baeea9a-ed64-4bf6-b9f8-ccf2351e07d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:42.176323 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6baeea9a-ed64-4bf6-b9f8-ccf2351e07d6 admin admin] PATH_INFO: `/v3/projects/7d45f968521b40bf8807ee060da82f1e/users/11b70bf7d4654cc9b5a92d11eaf659df/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:42.179530 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6baeea9a-ed64-4bf6-b9f8-ccf2351e07d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7d45f968521b40bf8807ee060da82f1e, user_id=11b70bf7d4654cc9b5a92d11eaf659df, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:42.184171 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6baeea9a-ed64-4bf6-b9f8-ccf2351e07d6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:42.217155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 696/1431] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:42 2022] PUT /identity/v3/projects/7d45f968521b40bf8807ee060da82f1e/users/11b70bf7d4654cc9b5a92d11eaf659df/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:42.222749 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 736/1432] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:42 2022] POST /identity/v3/users => generated 291 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:42.227548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6baeea9a-ed64-4bf6-b9f8-ccf2351e07d6 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:42.239332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5913aee-e446-4c22-9a8a-d58d033c4a5b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:42.240879 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5913aee-e446-4c22-9a8a-d58d033c4a5b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:42.241518 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5913aee-e446-4c22-9a8a-d58d033c4a5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:42.345577 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-f5913aee-e446-4c22-9a8a-d58d033c4a5b None None] MFA Rules not processed for user `11b70bf7d4654cc9b5a92d11eaf659df`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:42.347300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-f5913aee-e446-4c22-9a8a-d58d033c4a5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:42.483136 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6baeea9a-ed64-4bf6-b9f8-ccf2351e07d6 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:42.506298 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168b6d70-fa3f-4e79-8825-3655c1a31e45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:42.506991 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168b6d70-fa3f-4e79-8825-3655c1a31e45 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:42.511017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168b6d70-fa3f-4e79-8825-3655c1a31e45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:42.513922 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168b6d70-fa3f-4e79-8825-3655c1a31e45 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:42.514678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168b6d70-fa3f-4e79-8825-3655c1a31e45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:42.526081 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168b6d70-fa3f-4e79-8825-3655c1a31e45 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:42.582226 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 697/1433] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:42 2022] GET /identity/v3/roles => generated 1420 bytes in 355 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 00:57:42.584400 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168b6d70-fa3f-4e79-8825-3655c1a31e45 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:42.628365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168b6d70-fa3f-4e79-8825-3655c1a31e45 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:42.652794 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8132c404-f262-4db4-b539-a276dabea699 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:42.658985 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8132c404-f262-4db4-b539-a276dabea699 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:42.658985 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8132c404-f262-4db4-b539-a276dabea699 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:42.658985 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8132c404-f262-4db4-b539-a276dabea699 admin admin] PATH_INFO: `/v3/projects/8b9dcccdabca4bfba036e822ca020eff/users/b279e37de527448ead3587c7b507d9f6/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:42.659302 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 737/1434] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:42 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 426 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:42.659302 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5913aee-e446-4c22-9a8a-d58d033c4a5b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:42.734515 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8132c404-f262-4db4-b539-a276dabea699 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8b9dcccdabca4bfba036e822ca020eff, user_id=b279e37de527448ead3587c7b507d9f6, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:42.741095 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8132c404-f262-4db4-b539-a276dabea699 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:42.751833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:42.793231 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 698/1435] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:42 2022] PUT /identity/v3/projects/8b9dcccdabca4bfba036e822ca020eff/users/b279e37de527448ead3587c7b507d9f6/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:42.804180 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8132c404-f262-4db4-b539-a276dabea699 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:42.809246 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8132c404-f262-4db4-b539-a276dabea699 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:42.837065 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5913aee-e446-4c22-9a8a-d58d033c4a5b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:43.097908 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e66723-3551-4413-aa14-c5051f986335 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:43.098557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e66723-3551-4413-aa14-c5051f986335 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:43.103609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e66723-3551-4413-aa14-c5051f986335 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:43.104766 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e66723-3551-4413-aa14-c5051f986335 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:43.147599 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e66723-3551-4413-aa14-c5051f986335 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:43.149095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e66723-3551-4413-aa14-c5051f986335 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:43.149406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8132c404-f262-4db4-b539-a276dabea699 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:43.168940 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea08bc9-2970-4ab8-8a0b-ecb80c30d58c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:43.173483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea08bc9-2970-4ab8-8a0b-ecb80c30d58c service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:43.173483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea08bc9-2970-4ab8-8a0b-ecb80c30d58c service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:43.173483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea08bc9-2970-4ab8-8a0b-ecb80c30d58c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:43.202572 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 738/1436] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:42 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 545 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:43.203243 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e66723-3551-4413-aa14-c5051f986335 service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:43.227707 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:43.448928 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e66723-3551-4413-aa14-c5051f986335 service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:43.465516 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e63bc5f-d6f0-4a67-b4c6-d54f493e5e7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:43.467554 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e63bc5f-d6f0-4a67-b4c6-d54f493e5e7f admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:43.468278 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e63bc5f-d6f0-4a67-b4c6-d54f493e5e7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:43.468560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e63bc5f-d6f0-4a67-b4c6-d54f493e5e7f admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:43.471399 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e63bc5f-d6f0-4a67-b4c6-d54f493e5e7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:43.476037 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e63bc5f-d6f0-4a67-b4c6-d54f493e5e7f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:43.500986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 739/1437] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:43 2022] GET /identity/v3/roles => generated 1420 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:43.503110 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e63bc5f-d6f0-4a67-b4c6-d54f493e5e7f admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:43.535844 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e63bc5f-d6f0-4a67-b4c6-d54f493e5e7f admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:43.552526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea08bc9-2970-4ab8-8a0b-ecb80c30d58c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:43.553792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e244af2-22d1-4c9c-bfac-b3e7d41aa2c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:43.554508 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e244af2-22d1-4c9c-bfac-b3e7d41aa2c8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:43.554795 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e244af2-22d1-4c9c-bfac-b3e7d41aa2c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:43.556320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea08bc9-2970-4ab8-8a0b-ecb80c30d58c service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:43.560890 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e244af2-22d1-4c9c-bfac-b3e7d41aa2c8 admin admin] PATH_INFO: `/v3/projects/8b9dcccdabca4bfba036e822ca020eff/users/b279e37de527448ead3587c7b507d9f6/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:43.566624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e244af2-22d1-4c9c-bfac-b3e7d41aa2c8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8b9dcccdabca4bfba036e822ca020eff, user_id=b279e37de527448ead3587c7b507d9f6, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:43.577853 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e244af2-22d1-4c9c-bfac-b3e7d41aa2c8 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:43.590437 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 699/1438] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:42 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 797 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:43.628002 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 740/1439] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:43 2022] PUT /identity/v3/projects/8b9dcccdabca4bfba036e822ca020eff/users/b279e37de527448ead3587c7b507d9f6/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:43.638741 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f894490-76a1-4748-a441-7b038ce985b8 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:43.639096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f894490-76a1-4748-a441-7b038ce985b8 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:43.640081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f894490-76a1-4748-a441-7b038ce985b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:43.643345 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8769cb63-a0d8-417a-8ad7-f0f82d03a31e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:43.643646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8769cb63-a0d8-417a-8ad7-f0f82d03a31e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:43.643931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8769cb63-a0d8-417a-8ad7-f0f82d03a31e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:43.697957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-3f894490-76a1-4748-a441-7b038ce985b8 None None] MFA Rules not processed for user `b279e37de527448ead3587c7b507d9f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:43.699244 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-3f894490-76a1-4748-a441-7b038ce985b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:43.702419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-8769cb63-a0d8-417a-8ad7-f0f82d03a31e None None] MFA Rules not processed for user `f4eb1a640186411097cb1e3bb9e92020`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:43.705440 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8769cb63-a0d8-417a-8ad7-f0f82d03a31e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:43.944884 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 700/1440] 10.222.0.36 () {58 vars in 1015 bytes} [Sun Oct 16 00:57:43 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:43.945852 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8769cb63-a0d8-417a-8ad7-f0f82d03a31e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:43.952953 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8769cb63-a0d8-417a-8ad7-f0f82d03a31e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:43.964037 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 741/1441] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:43 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 326 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:43.978851 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f894490-76a1-4748-a441-7b038ce985b8 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:44.135532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8769cb63-a0d8-417a-8ad7-f0f82d03a31e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:44.145327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c990b334-99fc-4d77-8a1a-d5087eda596a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:44.146376 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c990b334-99fc-4d77-8a1a-d5087eda596a service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:44.146977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c990b334-99fc-4d77-8a1a-d5087eda596a service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:44.147363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c990b334-99fc-4d77-8a1a-d5087eda596a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:44.150095 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c990b334-99fc-4d77-8a1a-d5087eda596a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:44.172877 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:44.174040 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f894490-76a1-4748-a441-7b038ce985b8 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:44.191247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e148b6-3f97-4382-bf4e-8252df311ee9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:44.192030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e148b6-3f97-4382-bf4e-8252df311ee9 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:44.192314 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e148b6-3f97-4382-bf4e-8252df311ee9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:44.192591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e148b6-3f97-4382-bf4e-8252df311ee9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:44.223621 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e148b6-3f97-4382-bf4e-8252df311ee9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:44.225141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e148b6-3f97-4382-bf4e-8252df311ee9 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:44.255125 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 742/1442] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:43 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 278 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:44.258328 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e148b6-3f97-4382-bf4e-8252df311ee9 service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:44.286305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e148b6-3f97-4382-bf4e-8252df311ee9 service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:44.340012 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2816b01-49b1-415c-976d-fa2f96557d8c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:44.342560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2816b01-49b1-415c-976d-fa2f96557d8c service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:44.343277 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2816b01-49b1-415c-976d-fa2f96557d8c service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:44.343779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2816b01-49b1-415c-976d-fa2f96557d8c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:44.346532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c990b334-99fc-4d77-8a1a-d5087eda596a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:44.349608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c990b334-99fc-4d77-8a1a-d5087eda596a service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:44.456208 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2816b01-49b1-415c-976d-fa2f96557d8c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:44.459124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2816b01-49b1-415c-976d-fa2f96557d8c service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:44.469961 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 701/1443] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:43 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 525 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:44.518814 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:44.560043 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 743/1444] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:44 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:45.786848 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06472a06-b97a-4757-9995-a5230e1080b1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:45.787458 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06472a06-b97a-4757-9995-a5230e1080b1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:45.787541 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06472a06-b97a-4757-9995-a5230e1080b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:45.862093 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-06472a06-b97a-4757-9995-a5230e1080b1 None None] MFA Rules not processed for user `466f2f8ddd184a98bdb5e922818905cc`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:45.877922 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-06472a06-b97a-4757-9995-a5230e1080b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:46.123350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 702/1445] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:45 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 337 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:47.474219 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2816b01-49b1-415c-976d-fa2f96557d8c service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:47.476256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-a2816b01-49b1-415c-976d-fa2f96557d8c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:47.664904 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2816b01-49b1-415c-976d-fa2f96557d8c service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:47.683080 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db25c53-a3b4-40eb-837d-7cf36c4069ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:47.684037 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db25c53-a3b4-40eb-837d-7cf36c4069ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:47.684595 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db25c53-a3b4-40eb-837d-7cf36c4069ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:47.687821 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db25c53-a3b4-40eb-837d-7cf36c4069ff admin admin] PATH_INFO: `/v3/projects/e0fab4fb42b54a239f72b2696485c3b7` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:47.690984 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db25c53-a3b4-40eb-837d-7cf36c4069ff admin admin] RBAC: Authorizing `identity:delete_project(project_id=e0fab4fb42b54a239f72b2696485c3b7)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:47.693756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db25c53-a3b4-40eb-837d-7cf36c4069ff admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:47.958695 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3db25c53-a3b4-40eb-837d-7cf36c4069ff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e0fab4fb42b54a239f72b2696485c3b7', 'request_id': 'req-3db25c53-a3b4-40eb-837d-7cf36c4069ff'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:47.986913 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3db25c53-a3b4-40eb-837d-7cf36c4069ff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e0fab4fb42b54a239f72b2696485c3b7', 'request_id': 'req-3db25c53-a3b4-40eb-837d-7cf36c4069ff'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:48.034178 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3db25c53-a3b4-40eb-837d-7cf36c4069ff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e0fab4fb42b54a239f72b2696485c3b7', 'request_id': 'req-3db25c53-a3b4-40eb-837d-7cf36c4069ff'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:48.095898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3db25c53-a3b4-40eb-837d-7cf36c4069ff admin admin] The token cache is being invalidate because project e0fab4fb42b54a239f72b2696485c3b7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:57:48.096247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3db25c53-a3b4-40eb-837d-7cf36c4069ff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:48.103662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3db25c53-a3b4-40eb-837d-7cf36c4069ff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:48.108746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 744/1446] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:57:47 2022] DELETE /identity/v3/projects/e0fab4fb42b54a239f72b2696485c3b7 => generated 0 bytes in 632 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:48.109947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06472a06-b97a-4757-9995-a5230e1080b1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:48.135880 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06472a06-b97a-4757-9995-a5230e1080b1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:48.380731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1b3dd1-8ca0-4288-9bfb-d1b79ca1aa69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:48.386298 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1b3dd1-8ca0-4288-9bfb-d1b79ca1aa69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:48.386475 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1b3dd1-8ca0-4288-9bfb-d1b79ca1aa69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:48.390231 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1b3dd1-8ca0-4288-9bfb-d1b79ca1aa69 admin admin] PATH_INFO: `/v3/users/35966572a4b545a5871996616eb4b553` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:48.402204 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1b3dd1-8ca0-4288-9bfb-d1b79ca1aa69 admin admin] RBAC: Authorizing `identity:delete_user(user_id=35966572a4b545a5871996616eb4b553)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:48.421074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1b3dd1-8ca0-4288-9bfb-d1b79ca1aa69 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:48.616925 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-9f1b3dd1-8ca0-4288-9bfb-d1b79ca1aa69 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '35966572a4b545a5871996616eb4b553'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:48.623199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-9f1b3dd1-8ca0-4288-9bfb-d1b79ca1aa69 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '35966572a4b545a5871996616eb4b553'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:48.654302 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-9f1b3dd1-8ca0-4288-9bfb-d1b79ca1aa69 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '35966572a4b545a5871996616eb4b553'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:48.655503 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-9f1b3dd1-8ca0-4288-9bfb-d1b79ca1aa69 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '35966572a4b545a5871996616eb4b553'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:48.658800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:48.693253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-9f1b3dd1-8ca0-4288-9bfb-d1b79ca1aa69 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '35966572a4b545a5871996616eb4b553'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:48.739294 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 703/1447] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:57:48 2022] DELETE /identity/v3/users/35966572a4b545a5871996616eb4b553 => generated 0 bytes in 630 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:50.156598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db25c53-a3b4-40eb-837d-7cf36c4069ff admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:50.158359 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-3db25c53-a3b4-40eb-837d-7cf36c4069ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:50.192817 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db25c53-a3b4-40eb-837d-7cf36c4069ff admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:50.533041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53938a33-f980-428d-bf38-0ea8963a1991 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:50.534976 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53938a33-f980-428d-bf38-0ea8963a1991 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:50.535428 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53938a33-f980-428d-bf38-0ea8963a1991 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:50.535865 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53938a33-f980-428d-bf38-0ea8963a1991 admin admin] PATH_INFO: `/v3/users/480e0ce54ba04565999cb976c0b7d814` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:50.539217 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53938a33-f980-428d-bf38-0ea8963a1991 admin admin] RBAC: Authorizing `identity:delete_user(user_id=480e0ce54ba04565999cb976c0b7d814)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:50.541448 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53938a33-f980-428d-bf38-0ea8963a1991 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:50.676378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:50.715984 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-53938a33-f980-428d-bf38-0ea8963a1991 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '480e0ce54ba04565999cb976c0b7d814'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:50.721651 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-53938a33-f980-428d-bf38-0ea8963a1991 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '480e0ce54ba04565999cb976c0b7d814'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:50.756872 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-53938a33-f980-428d-bf38-0ea8963a1991 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '480e0ce54ba04565999cb976c0b7d814'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:50.828272 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-53938a33-f980-428d-bf38-0ea8963a1991 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '480e0ce54ba04565999cb976c0b7d814'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:50.864110 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-53938a33-f980-428d-bf38-0ea8963a1991 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '480e0ce54ba04565999cb976c0b7d814'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:50.864575 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1b3dd1-8ca0-4288-9bfb-d1b79ca1aa69 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:50.866832 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9f1b3dd1-8ca0-4288-9bfb-d1b79ca1aa69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:50.905893 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 745/1448] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:57:50 2022] DELETE /identity/v3/users/480e0ce54ba04565999cb976c0b7d814 => generated 0 bytes in 745 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:50.907462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fda3f12-ff27-4fd5-9ae9-1baf0a4898c5 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:50.907756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fda3f12-ff27-4fd5-9ae9-1baf0a4898c5 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:50.908137 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fda3f12-ff27-4fd5-9ae9-1baf0a4898c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:50.923680 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1b3dd1-8ca0-4288-9bfb-d1b79ca1aa69 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:51.020153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-0fda3f12-ff27-4fd5-9ae9-1baf0a4898c5 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:51.021469 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-0fda3f12-ff27-4fd5-9ae9-1baf0a4898c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:51.165662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771d6079-3505-494e-97f7-1fb1c442efba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:51.166373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771d6079-3505-494e-97f7-1fb1c442efba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:51.169869 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771d6079-3505-494e-97f7-1fb1c442efba admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:51.170329 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771d6079-3505-494e-97f7-1fb1c442efba admin admin] PATH_INFO: `/v3/projects/64bd21611c864051a08b910c68d58539` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:51.173552 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771d6079-3505-494e-97f7-1fb1c442efba admin admin] RBAC: Authorizing `identity:delete_project(project_id=64bd21611c864051a08b910c68d58539)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:51.180590 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771d6079-3505-494e-97f7-1fb1c442efba admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:51.296682 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 746/1449] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 390 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:51.306318 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fda3f12-ff27-4fd5-9ae9-1baf0a4898c5 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:51.308235 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-0fda3f12-ff27-4fd5-9ae9-1baf0a4898c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:51.353904 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fda3f12-ff27-4fd5-9ae9-1baf0a4898c5 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:51.363802 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-771d6079-3505-494e-97f7-1fb1c442efba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '64bd21611c864051a08b910c68d58539', 'request_id': 'req-771d6079-3505-494e-97f7-1fb1c442efba'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:51.390331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-771d6079-3505-494e-97f7-1fb1c442efba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '64bd21611c864051a08b910c68d58539', 'request_id': 'req-771d6079-3505-494e-97f7-1fb1c442efba'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:51.415632 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-771d6079-3505-494e-97f7-1fb1c442efba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '64bd21611c864051a08b910c68d58539', 'request_id': 'req-771d6079-3505-494e-97f7-1fb1c442efba'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:51.454047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-771d6079-3505-494e-97f7-1fb1c442efba admin admin] The token cache is being invalidate because project 64bd21611c864051a08b910c68d58539 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:57:51.454393 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-771d6079-3505-494e-97f7-1fb1c442efba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:51.455930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-771d6079-3505-494e-97f7-1fb1c442efba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:51.463405 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 704/1450] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:57:50 2022] DELETE /identity/v3/projects/64bd21611c864051a08b910c68d58539 => generated 0 bytes in 599 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:57:51.464445 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771d6079-3505-494e-97f7-1fb1c442efba admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:51.467629 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-771d6079-3505-494e-97f7-1fb1c442efba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:51.535954 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e230b0bb-93a9-4d5c-95d3-26c3449e4720 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:51.537050 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e230b0bb-93a9-4d5c-95d3-26c3449e4720 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:51.537767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e230b0bb-93a9-4d5c-95d3-26c3449e4720 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:51.538300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e230b0bb-93a9-4d5c-95d3-26c3449e4720 admin admin] PATH_INFO: `/v3/users/8a516b2896424ca7a3169d667ab3b8b3` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:51.541606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e230b0bb-93a9-4d5c-95d3-26c3449e4720 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8a516b2896424ca7a3169d667ab3b8b3)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:51.546973 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e230b0bb-93a9-4d5c-95d3-26c3449e4720 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:51.706263 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771d6079-3505-494e-97f7-1fb1c442efba admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:51.712806 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ee7c5a-0a38-4211-93a2-38e24659e686 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:51.713965 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e230b0bb-93a9-4d5c-95d3-26c3449e4720 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8a516b2896424ca7a3169d667ab3b8b3'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:51.714131 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ee7c5a-0a38-4211-93a2-38e24659e686 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:51.714403 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ee7c5a-0a38-4211-93a2-38e24659e686 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:51.714700 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ee7c5a-0a38-4211-93a2-38e24659e686 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:51.716770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e230b0bb-93a9-4d5c-95d3-26c3449e4720 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8a516b2896424ca7a3169d667ab3b8b3'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:51.718628 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c9ee7c5a-0a38-4211-93a2-38e24659e686 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:51.723468 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:51.741120 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e230b0bb-93a9-4d5c-95d3-26c3449e4720 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8a516b2896424ca7a3169d667ab3b8b3'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:51.789776 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ee7c5a-0a38-4211-93a2-38e24659e686 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:51.799431 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ee7c5a-0a38-4211-93a2-38e24659e686 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:51.800563 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e230b0bb-93a9-4d5c-95d3-26c3449e4720 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8a516b2896424ca7a3169d667ab3b8b3'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:51.835144 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e230b0bb-93a9-4d5c-95d3-26c3449e4720 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8a516b2896424ca7a3169d667ab3b8b3'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:51.851297 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 705/1451] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:51.855156 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440da4f8-00ff-4790-b3da-797054b0d677 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:51.858788 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440da4f8-00ff-4790-b3da-797054b0d677 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:51.859637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440da4f8-00ff-4790-b3da-797054b0d677 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:51.867963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 747/1452] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:57:51 2022] DELETE /identity/v3/users/8a516b2896424ca7a3169d667ab3b8b3 => generated 0 bytes in 562 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:57:51.875929 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2776b1e2-77e1-4259-85ac-666a166cbb60 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:51.877324 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2776b1e2-77e1-4259-85ac-666a166cbb60 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:51.880096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2776b1e2-77e1-4259-85ac-666a166cbb60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:51.923789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:51.951905 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-440da4f8-00ff-4790-b3da-797054b0d677 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:51.955546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-440da4f8-00ff-4790-b3da-797054b0d677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:51.961215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-2776b1e2-77e1-4259-85ac-666a166cbb60 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:51.964164 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-2776b1e2-77e1-4259-85ac-666a166cbb60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:52.085913 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 748/1453] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:52.187534 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 706/1454] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 328 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:52.193456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c22c519-2ec5-498f-b8de-3c83c911dfcf None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:52.194240 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c22c519-2ec5-498f-b8de-3c83c911dfcf None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:52.194730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c22c519-2ec5-498f-b8de-3c83c911dfcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:52.210935 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440da4f8-00ff-4790-b3da-797054b0d677 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:52.214121 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-440da4f8-00ff-4790-b3da-797054b0d677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:52.276392 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-4c22c519-2ec5-498f-b8de-3c83c911dfcf None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:52.289309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4c22c519-2ec5-498f-b8de-3c83c911dfcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:52.343267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 749/1455] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:52.345876 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c22c519-2ec5-498f-b8de-3c83c911dfcf None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:52.374543 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c22c519-2ec5-498f-b8de-3c83c911dfcf None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:52.391065 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdd31ab-aad2-4b50-a637-02b085f485e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:52.392237 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdd31ab-aad2-4b50-a637-02b085f485e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:52.392535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdd31ab-aad2-4b50-a637-02b085f485e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:52.392815 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdd31ab-aad2-4b50-a637-02b085f485e2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:52.393489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdd31ab-aad2-4b50-a637-02b085f485e2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:57:52.395880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdd31ab-aad2-4b50-a637-02b085f485e2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:52.402926 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdd31ab-aad2-4b50-a637-02b085f485e2 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:52.427192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440da4f8-00ff-4790-b3da-797054b0d677 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:52.451763 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 750/1456] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:57:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:57:52.455007 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28430205-0a35-4f1b-b849-77f10cbcd57d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:52.455985 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28430205-0a35-4f1b-b849-77f10cbcd57d service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:52.456380 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28430205-0a35-4f1b-b849-77f10cbcd57d service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:52.456942 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28430205-0a35-4f1b-b849-77f10cbcd57d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:52.461469 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdd31ab-aad2-4b50-a637-02b085f485e2 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:52.493708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdd31ab-aad2-4b50-a637-02b085f485e2 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:52.511658 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28430205-0a35-4f1b-b849-77f10cbcd57d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:52.513026 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a460ac8-c5f4-4bd7-8537-8dc6590375c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:52.513398 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28430205-0a35-4f1b-b849-77f10cbcd57d service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:52.514083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a460ac8-c5f4-4bd7-8537-8dc6590375c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:52.514551 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a460ac8-c5f4-4bd7-8537-8dc6590375c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:52.515045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a460ac8-c5f4-4bd7-8537-8dc6590375c2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:52.515861 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a460ac8-c5f4-4bd7-8537-8dc6590375c2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:52.517947 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a460ac8-c5f4-4bd7-8537-8dc6590375c2 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:52.553131 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 707/1457] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:52.566794 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 751/1458] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:52 2022] POST /identity/v3/projects => generated 371 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:52.570735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28430205-0a35-4f1b-b849-77f10cbcd57d service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:52.601734 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28430205-0a35-4f1b-b849-77f10cbcd57d service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:52.785371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09fc465-ece3-479f-8fae-c6bcff64fcd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:52.786163 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09fc465-ece3-479f-8fae-c6bcff64fcd3 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:52.786554 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09fc465-ece3-479f-8fae-c6bcff64fcd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:52.786947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09fc465-ece3-479f-8fae-c6bcff64fcd3 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:52.787807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09fc465-ece3-479f-8fae-c6bcff64fcd3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:52.789912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09fc465-ece3-479f-8fae-c6bcff64fcd3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:52.868421 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 708/1459] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:52 2022] POST /identity/v3/users => generated 286 bytes in 298 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:52.872524 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a460ac8-c5f4-4bd7-8537-8dc6590375c2 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:52.892953 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a460ac8-c5f4-4bd7-8537-8dc6590375c2 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:52.910083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aeecc3b-a221-46f8-8969-56aebf3e5c2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:52.915355 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aeecc3b-a221-46f8-8969-56aebf3e5c2d admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:52.916575 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aeecc3b-a221-46f8-8969-56aebf3e5c2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:52.917296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aeecc3b-a221-46f8-8969-56aebf3e5c2d admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:52.921597 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aeecc3b-a221-46f8-8969-56aebf3e5c2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:52.932126 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aeecc3b-a221-46f8-8969-56aebf3e5c2d admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:52.988566 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 752/1460] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:52 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:52.992243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09fc465-ece3-479f-8fae-c6bcff64fcd3 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:53.026964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09fc465-ece3-479f-8fae-c6bcff64fcd3 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:53.034032 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aeecc3b-a221-46f8-8969-56aebf3e5c2d admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:53.035875 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-3aeecc3b-a221-46f8-8969-56aebf3e5c2d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:53.047672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67df0feb-c17c-44c1-8b95-62f1d6c103eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:53.048642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67df0feb-c17c-44c1-8b95-62f1d6c103eb admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:53.049373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67df0feb-c17c-44c1-8b95-62f1d6c103eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:53.049939 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67df0feb-c17c-44c1-8b95-62f1d6c103eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:53.050931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67df0feb-c17c-44c1-8b95-62f1d6c103eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:53.053079 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67df0feb-c17c-44c1-8b95-62f1d6c103eb admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:53.062738 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aeecc3b-a221-46f8-8969-56aebf3e5c2d admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:53.072895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 709/1461] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:52 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 00:57:53.085479 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67df0feb-c17c-44c1-8b95-62f1d6c103eb admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:53.101529 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815676d9-16c9-4eac-b220-0176fc5a3038 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:53.103742 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815676d9-16c9-4eac-b220-0176fc5a3038 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:53.104821 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815676d9-16c9-4eac-b220-0176fc5a3038 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:53.108033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815676d9-16c9-4eac-b220-0176fc5a3038 admin admin] PATH_INFO: `/v3/users/193e6edd8c334e428fe5aa8235567b80` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:53.110143 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67df0feb-c17c-44c1-8b95-62f1d6c103eb admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:53.110995 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815676d9-16c9-4eac-b220-0176fc5a3038 admin admin] RBAC: Authorizing `identity:delete_user(user_id=193e6edd8c334e428fe5aa8235567b80)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:53.125255 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24facc9e-4c91-4d5f-9ba6-de711bf63de1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:53.126365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24facc9e-4c91-4d5f-9ba6-de711bf63de1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:53.126818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24facc9e-4c91-4d5f-9ba6-de711bf63de1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:53.127358 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24facc9e-4c91-4d5f-9ba6-de711bf63de1 admin admin] PATH_INFO: `/v3/projects/0d2920f753864a4c807705e2a33d0779/users/42476c74af934f3a9f7c5124defd55fa/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:53.140774 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815676d9-16c9-4eac-b220-0176fc5a3038 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:53.152282 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24facc9e-4c91-4d5f-9ba6-de711bf63de1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d2920f753864a4c807705e2a33d0779, user_id=42476c74af934f3a9f7c5124defd55fa, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:53.156713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24facc9e-4c91-4d5f-9ba6-de711bf63de1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:53.197672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 710/1462] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:53 2022] PUT /identity/v3/projects/0d2920f753864a4c807705e2a33d0779/users/42476c74af934f3a9f7c5124defd55fa/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:53.210229 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94857a7d-fd6e-427c-9dbc-894f9c8a2a63 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:53.210802 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94857a7d-fd6e-427c-9dbc-894f9c8a2a63 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:53.211315 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94857a7d-fd6e-427c-9dbc-894f9c8a2a63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:53.244721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:53.262189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-815676d9-16c9-4eac-b220-0176fc5a3038 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '193e6edd8c334e428fe5aa8235567b80'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:53.265136 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-815676d9-16c9-4eac-b220-0176fc5a3038 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '193e6edd8c334e428fe5aa8235567b80'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:53.287359 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-94857a7d-fd6e-427c-9dbc-894f9c8a2a63 None None] MFA Rules not processed for user `42476c74af934f3a9f7c5124defd55fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:53.289662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-94857a7d-fd6e-427c-9dbc-894f9c8a2a63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:53.290406 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-815676d9-16c9-4eac-b220-0176fc5a3038 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '193e6edd8c334e428fe5aa8235567b80'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:53.349283 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-815676d9-16c9-4eac-b220-0176fc5a3038 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '193e6edd8c334e428fe5aa8235567b80'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:53.351776 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:53.369338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-815676d9-16c9-4eac-b220-0176fc5a3038 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '193e6edd8c334e428fe5aa8235567b80'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:53.392267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 753/1463] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:57:53 2022] DELETE /identity/v3/users/193e6edd8c334e428fe5aa8235567b80 => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:53.397726 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dea1eb-5249-4e7d-b3c0-1dcc1ef7a4eb None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:53.409148 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dea1eb-5249-4e7d-b3c0-1dcc1ef7a4eb None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:53.410554 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dea1eb-5249-4e7d-b3c0-1dcc1ef7a4eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:53.484341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-63dea1eb-5249-4e7d-b3c0-1dcc1ef7a4eb None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:53.488833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-63dea1eb-5249-4e7d-b3c0-1dcc1ef7a4eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:53.502550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 711/1464] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:53 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:57:53.503767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94857a7d-fd6e-427c-9dbc-894f9c8a2a63 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:53.505314 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-94857a7d-fd6e-427c-9dbc-894f9c8a2a63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:53.726602 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94857a7d-fd6e-427c-9dbc-894f9c8a2a63 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:53.838836 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 754/1465] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 443 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:53.908970 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63dea1eb-5249-4e7d-b3c0-1dcc1ef7a4eb None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:53.910765 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-63dea1eb-5249-4e7d-b3c0-1dcc1ef7a4eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:53.958010 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64863850-b7c9-4756-85c6-e35f948e0728 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:53.958712 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64863850-b7c9-4756-85c6-e35f948e0728 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:53.959125 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64863850-b7c9-4756-85c6-e35f948e0728 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:53.959413 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64863850-b7c9-4756-85c6-e35f948e0728 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:53.960076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64863850-b7c9-4756-85c6-e35f948e0728 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:53.966477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64863850-b7c9-4756-85c6-e35f948e0728 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:54.073256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 712/1466] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:53 2022] POST /identity/v3/projects => generated 371 bytes in 570 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:54.083464 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64863850-b7c9-4756-85c6-e35f948e0728 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:54.140151 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63dea1eb-5249-4e7d-b3c0-1dcc1ef7a4eb None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:54.212244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:54.409279 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64863850-b7c9-4756-85c6-e35f948e0728 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:54.428967 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffff2e91-3063-4520-ac95-8979e5f66703 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:54.429497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf050f88-8cac-4d82-8564-8008a91ae4ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:54.430651 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffff2e91-3063-4520-ac95-8979e5f66703 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:54.430958 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf050f88-8cac-4d82-8564-8008a91ae4ab admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:54.431598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffff2e91-3063-4520-ac95-8979e5f66703 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:54.431963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf050f88-8cac-4d82-8564-8008a91ae4ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:54.432429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf050f88-8cac-4d82-8564-8008a91ae4ab admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:54.432701 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffff2e91-3063-4520-ac95-8979e5f66703 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:54.433728 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf050f88-8cac-4d82-8564-8008a91ae4ab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:54.442071 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf050f88-8cac-4d82-8564-8008a91ae4ab admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:54.473160 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffff2e91-3063-4520-ac95-8979e5f66703 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:54.476707 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffff2e91-3063-4520-ac95-8979e5f66703 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:54.521099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 755/1467] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 612 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:57:54.521293 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 713/1468] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:54 2022] POST /identity/v3/users => generated 292 bytes in 438 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:54.526586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffff2e91-3063-4520-ac95-8979e5f66703 service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:54.546824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffff2e91-3063-4520-ac95-8979e5f66703 service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:54.554182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edc9088-0744-472d-9356-f082ea532bbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:54.555057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edc9088-0744-472d-9356-f082ea532bbf admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:54.555564 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edc9088-0744-472d-9356-f082ea532bbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:54.556215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edc9088-0744-472d-9356-f082ea532bbf admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:54.557378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edc9088-0744-472d-9356-f082ea532bbf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:54.561213 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edc9088-0744-472d-9356-f082ea532bbf admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:54.581518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7f9c7f-847e-4a0d-b504-cb4c5790a7af None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:54.581518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7f9c7f-847e-4a0d-b504-cb4c5790a7af None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:54.582472 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7f9c7f-847e-4a0d-b504-cb4c5790a7af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:54.608745 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 756/1469] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:54 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:54.618647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edc9088-0744-472d-9356-f082ea532bbf admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:54.638844 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-be7f9c7f-847e-4a0d-b504-cb4c5790a7af None None] MFA Rules not processed for user `ccece85a85fb48c4887b52c96c5bc642`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:54.640308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-be7f9c7f-847e-4a0d-b504-cb4c5790a7af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:54.650328 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edc9088-0744-472d-9356-f082ea532bbf admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:54.663552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cdbb0a-7968-4a31-998b-4e3fc375b625 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:54.665293 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cdbb0a-7968-4a31-998b-4e3fc375b625 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:54.665647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cdbb0a-7968-4a31-998b-4e3fc375b625 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:54.665934 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cdbb0a-7968-4a31-998b-4e3fc375b625 admin admin] PATH_INFO: `/v3/projects/8540cf30cae44346aef35f64936213df/users/4b8fe79c527345e9a8d072b30e9ba349/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:54.692802 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cdbb0a-7968-4a31-998b-4e3fc375b625 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8540cf30cae44346aef35f64936213df, user_id=4b8fe79c527345e9a8d072b30e9ba349, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:54.723668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cdbb0a-7968-4a31-998b-4e3fc375b625 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:54.804713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 714/1470] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:54 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:54.806311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 757/1471] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:54 2022] PUT /identity/v3/projects/8540cf30cae44346aef35f64936213df/users/4b8fe79c527345e9a8d072b30e9ba349/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:54.813001 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7f9c7f-847e-4a0d-b504-cb4c5790a7af None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:54.817562 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faee518f-ae11-47f7-917e-415d1092d261 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:54.817865 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faee518f-ae11-47f7-917e-415d1092d261 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:54.818422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faee518f-ae11-47f7-917e-415d1092d261 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:54.827824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 758/1472] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 16 00:57:54 2022] GET /identity => generated 270 bytes in 21 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:57:54.852201 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7f9c7f-847e-4a0d-b504-cb4c5790a7af None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:54.980715 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d63bdd6-d51f-47d7-ae88-e8871d4ca8f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:54.982817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d63bdd6-d51f-47d7-ae88-e8871d4ca8f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:54.983579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d63bdd6-d51f-47d7-ae88-e8871d4ca8f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:54.984126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d63bdd6-d51f-47d7-ae88-e8871d4ca8f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:55.003181 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d63bdd6-d51f-47d7-ae88-e8871d4ca8f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:55.004719 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc08a1a5-c1ea-4afe-b2f8-3795f1ae8a60 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:55.005135 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc08a1a5-c1ea-4afe-b2f8-3795f1ae8a60 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:55.005424 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc08a1a5-c1ea-4afe-b2f8-3795f1ae8a60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:55.006183 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d63bdd6-d51f-47d7-ae88-e8871d4ca8f3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:55.048572 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-cc08a1a5-c1ea-4afe-b2f8-3795f1ae8a60 None None] MFA Rules not processed for user `4eba26813e154595a12a67fe4be5eeff`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:55.050231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-cc08a1a5-c1ea-4afe-b2f8-3795f1ae8a60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:55.053821 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 715/1473] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:57:54 2022] GET /identity/v3/roles => generated 1420 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:57:55.055092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d63bdd6-d51f-47d7-ae88-e8871d4ca8f3 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:55.078474 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:55.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d63bdd6-d51f-47d7-ae88-e8871d4ca8f3 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:55.108407 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a8a0bd-f565-4de4-b02b-badc6bef3581 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:55.109060 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a8a0bd-f565-4de4-b02b-badc6bef3581 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:55.109347 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a8a0bd-f565-4de4-b02b-badc6bef3581 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:55.109626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a8a0bd-f565-4de4-b02b-badc6bef3581 admin admin] PATH_INFO: `/v3/projects/8540cf30cae44346aef35f64936213df/users/4b8fe79c527345e9a8d072b30e9ba349/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:55.126393 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a8a0bd-f565-4de4-b02b-badc6bef3581 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8540cf30cae44346aef35f64936213df, user_id=4b8fe79c527345e9a8d072b30e9ba349, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:55.130424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a8a0bd-f565-4de4-b02b-badc6bef3581 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:55.192146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 716/1474] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:57:55 2022] PUT /identity/v3/projects/8540cf30cae44346aef35f64936213df/users/4b8fe79c527345e9a8d072b30e9ba349/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:55.207925 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285e9bc9-0987-4a14-91e5-4d2d390123de None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:55.208554 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285e9bc9-0987-4a14-91e5-4d2d390123de None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:55.208924 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285e9bc9-0987-4a14-91e5-4d2d390123de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:55.260189 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-285e9bc9-0987-4a14-91e5-4d2d390123de None None] MFA Rules not processed for user `4b8fe79c527345e9a8d072b30e9ba349`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:55.262804 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-285e9bc9-0987-4a14-91e5-4d2d390123de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:55.264575 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 759/1475] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 16 00:57:55 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:55.363219 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc08a1a5-c1ea-4afe-b2f8-3795f1ae8a60 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:55.365284 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-cc08a1a5-c1ea-4afe-b2f8-3795f1ae8a60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:55.390042 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc08a1a5-c1ea-4afe-b2f8-3795f1ae8a60 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:55.401533 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:55.493006 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 717/1476] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:55 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 286 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:57:55.512033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285e9bc9-0987-4a14-91e5-4d2d390123de None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:55.514940 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-285e9bc9-0987-4a14-91e5-4d2d390123de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:55.550563 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34a3edf-c57e-476c-9665-9f746f08fe62 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:55.552453 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34a3edf-c57e-476c-9665-9f746f08fe62 service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:55.552901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34a3edf-c57e-476c-9665-9f746f08fe62 service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:55.553184 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34a3edf-c57e-476c-9665-9f746f08fe62 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:55.585212 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34a3edf-c57e-476c-9665-9f746f08fe62 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:55.590338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34a3edf-c57e-476c-9665-9f746f08fe62 service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:55.604739 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285e9bc9-0987-4a14-91e5-4d2d390123de None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:55.616220 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae184cba-f9e0-4116-9dac-5de3e41d64b5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:55.617493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae184cba-f9e0-4116-9dac-5de3e41d64b5 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:55.617998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae184cba-f9e0-4116-9dac-5de3e41d64b5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:55.618634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae184cba-f9e0-4116-9dac-5de3e41d64b5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:55.636056 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae184cba-f9e0-4116-9dac-5de3e41d64b5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:55.639356 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae184cba-f9e0-4116-9dac-5de3e41d64b5 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:55.645543 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 760/1477] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 284 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:57:55.668067 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 718/1478] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:55 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:56.009472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34a3edf-c57e-476c-9665-9f746f08fe62 service nova] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:56.014053 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-e34a3edf-c57e-476c-9665-9f746f08fe62 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:56.148637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34a3edf-c57e-476c-9665-9f746f08fe62 service nova] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:56.177003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:56.177791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:56.178076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:56.178353 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65 admin admin] PATH_INFO: `/v3/projects/1e231fb307c24d1897d19a612aaa1259` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:56.183386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1e231fb307c24d1897d19a612aaa1259)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:56.189359 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:56.228813 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae184cba-f9e0-4116-9dac-5de3e41d64b5 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:56.230334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ae184cba-f9e0-4116-9dac-5de3e41d64b5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:56.259532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae184cba-f9e0-4116-9dac-5de3e41d64b5 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:56.272022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b987cf-3126-48e2-9154-a6591a6609c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:56.272689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b987cf-3126-48e2-9154-a6591a6609c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:56.272972 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b987cf-3126-48e2-9154-a6591a6609c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:56.273245 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b987cf-3126-48e2-9154-a6591a6609c0 admin admin] PATH_INFO: `/v3/projects/ebf01e91c24e4380a3115bf63dda7bbd` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:56.274520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b987cf-3126-48e2-9154-a6591a6609c0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ebf01e91c24e4380a3115bf63dda7bbd)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:56.276436 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b987cf-3126-48e2-9154-a6591a6609c0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:56.341945 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:56.384992 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1e231fb307c24d1897d19a612aaa1259', 'request_id': 'req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.411920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1e231fb307c24d1897d19a612aaa1259', 'request_id': 'req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.416408 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-87b987cf-3126-48e2-9154-a6591a6609c0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ebf01e91c24e4380a3115bf63dda7bbd', 'request_id': 'req-87b987cf-3126-48e2-9154-a6591a6609c0'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.436356 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-87b987cf-3126-48e2-9154-a6591a6609c0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ebf01e91c24e4380a3115bf63dda7bbd', 'request_id': 'req-87b987cf-3126-48e2-9154-a6591a6609c0'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.439457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1e231fb307c24d1897d19a612aaa1259', 'request_id': 'req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.448838 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-87b987cf-3126-48e2-9154-a6591a6609c0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ebf01e91c24e4380a3115bf63dda7bbd', 'request_id': 'req-87b987cf-3126-48e2-9154-a6591a6609c0'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.469147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-87b987cf-3126-48e2-9154-a6591a6609c0 admin admin] The token cache is being invalidate because project ebf01e91c24e4380a3115bf63dda7bbd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:57:56.469743 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-87b987cf-3126-48e2-9154-a6591a6609c0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.470765 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-87b987cf-3126-48e2-9154-a6591a6609c0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.472842 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 719/1479] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:57:56 2022] DELETE /identity/v3/projects/ebf01e91c24e4380a3115bf63dda7bbd => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:56.473986 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b987cf-3126-48e2-9154-a6591a6609c0 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:56.475680 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-87b987cf-3126-48e2-9154-a6591a6609c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:56.483405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65 admin admin] The token cache is being invalidate because project 1e231fb307c24d1897d19a612aaa1259 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:57:56.483405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.487447 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.494269 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 761/1480] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:57:56 2022] DELETE /identity/v3/projects/1e231fb307c24d1897d19a612aaa1259 => generated 0 bytes in 485 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:56.508783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:56.510415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:56.605191 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:56.618385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b987cf-3126-48e2-9154-a6591a6609c0 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:56.626576 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72340a03-3b78-4b11-a180-3983cd07696a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:56.627977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72340a03-3b78-4b11-a180-3983cd07696a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:56.628277 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72340a03-3b78-4b11-a180-3983cd07696a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:56.628614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72340a03-3b78-4b11-a180-3983cd07696a admin admin] PATH_INFO: `/v3/users/42476c74af934f3a9f7c5124defd55fa` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:56.630080 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72340a03-3b78-4b11-a180-3983cd07696a admin admin] RBAC: Authorizing `identity:delete_user(user_id=42476c74af934f3a9f7c5124defd55fa)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:56.633485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72340a03-3b78-4b11-a180-3983cd07696a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:56.725004 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-72340a03-3b78-4b11-a180-3983cd07696a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '42476c74af934f3a9f7c5124defd55fa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.725770 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-72340a03-3b78-4b11-a180-3983cd07696a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '42476c74af934f3a9f7c5124defd55fa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.741510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-72340a03-3b78-4b11-a180-3983cd07696a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '42476c74af934f3a9f7c5124defd55fa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.742409 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-72340a03-3b78-4b11-a180-3983cd07696a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '42476c74af934f3a9f7c5124defd55fa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.743543 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19f2bce-ccc7-4bfb-9ea8-3f9e6aee7d65 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:56.750987 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd550846-42c7-4961-8238-aeb24aac9849 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:56.751850 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd550846-42c7-4961-8238-aeb24aac9849 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:56.752314 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd550846-42c7-4961-8238-aeb24aac9849 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:56.752680 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd550846-42c7-4961-8238-aeb24aac9849 admin admin] PATH_INFO: `/v3/users/b279e37de527448ead3587c7b507d9f6` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:56.755067 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd550846-42c7-4961-8238-aeb24aac9849 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b279e37de527448ead3587c7b507d9f6)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:56.757329 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-72340a03-3b78-4b11-a180-3983cd07696a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '42476c74af934f3a9f7c5124defd55fa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.758550 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd550846-42c7-4961-8238-aeb24aac9849 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:56.773014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 720/1481] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:57:56 2022] DELETE /identity/v3/users/42476c74af934f3a9f7c5124defd55fa => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:56.774836 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72340a03-3b78-4b11-a180-3983cd07696a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:56.776442 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-72340a03-3b78-4b11-a180-3983cd07696a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:56.858784 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-bd550846-42c7-4961-8238-aeb24aac9849 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b279e37de527448ead3587c7b507d9f6'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.859646 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-bd550846-42c7-4961-8238-aeb24aac9849 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b279e37de527448ead3587c7b507d9f6'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.872881 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-bd550846-42c7-4961-8238-aeb24aac9849 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b279e37de527448ead3587c7b507d9f6'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.910051 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72340a03-3b78-4b11-a180-3983cd07696a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:56.916621 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-bd550846-42c7-4961-8238-aeb24aac9849 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b279e37de527448ead3587c7b507d9f6'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.927788 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7091d3c-310b-4d39-aca3-37383356ec7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:56.932732 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7091d3c-310b-4d39-aca3-37383356ec7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:56.932732 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7091d3c-310b-4d39-aca3-37383356ec7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:56.933470 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7091d3c-310b-4d39-aca3-37383356ec7f admin admin] PATH_INFO: `/v3/users/11b70bf7d4654cc9b5a92d11eaf659df` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:56.936951 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7091d3c-310b-4d39-aca3-37383356ec7f admin admin] RBAC: Authorizing `identity:delete_user(user_id=11b70bf7d4654cc9b5a92d11eaf659df)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:56.938567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-bd550846-42c7-4961-8238-aeb24aac9849 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b279e37de527448ead3587c7b507d9f6'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:56.944683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7091d3c-310b-4d39-aca3-37383356ec7f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:56.948477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 762/1482] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:57:56 2022] DELETE /identity/v3/users/b279e37de527448ead3587c7b507d9f6 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:56.950193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd550846-42c7-4961-8238-aeb24aac9849 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:56.963521 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-bd550846-42c7-4961-8238-aeb24aac9849 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:57.064034 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d7091d3c-310b-4d39-aca3-37383356ec7f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '11b70bf7d4654cc9b5a92d11eaf659df'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:57.065461 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d7091d3c-310b-4d39-aca3-37383356ec7f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '11b70bf7d4654cc9b5a92d11eaf659df'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:57.066749 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:57.078471 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d7091d3c-310b-4d39-aca3-37383356ec7f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '11b70bf7d4654cc9b5a92d11eaf659df'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:57.079283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d7091d3c-310b-4d39-aca3-37383356ec7f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '11b70bf7d4654cc9b5a92d11eaf659df'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:57.098694 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d7091d3c-310b-4d39-aca3-37383356ec7f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '11b70bf7d4654cc9b5a92d11eaf659df'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:57.104458 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:57.119260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 721/1483] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:57:56 2022] DELETE /identity/v3/users/11b70bf7d4654cc9b5a92d11eaf659df => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:57.120596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd550846-42c7-4961-8238-aeb24aac9849 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:57.126394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90e30c2-8c18-40c4-8b0c-289964eb88ae None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:57.126764 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90e30c2-8c18-40c4-8b0c-289964eb88ae None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:57.128119 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90e30c2-8c18-40c4-8b0c-289964eb88ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:57.182008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e90e30c2-8c18-40c4-8b0c-289964eb88ae None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:57.184117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e90e30c2-8c18-40c4-8b0c-289964eb88ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:57.271351 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60ad173-9503-4471-9d30-33268d468f67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:57.272115 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60ad173-9503-4471-9d30-33268d468f67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:57.272475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60ad173-9503-4471-9d30-33268d468f67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:57.272760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60ad173-9503-4471-9d30-33268d468f67 admin admin] PATH_INFO: `/v3/projects/42941d5f4dae45c6b1cfa5a803a943d6` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:57.274308 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60ad173-9503-4471-9d30-33268d468f67 admin admin] RBAC: Authorizing `identity:delete_project(project_id=42941d5f4dae45c6b1cfa5a803a943d6)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:57.276589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60ad173-9503-4471-9d30-33268d468f67 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:57.284557 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 722/1484] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:57.318293 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90e30c2-8c18-40c4-8b0c-289964eb88ae None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:57.322618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e90e30c2-8c18-40c4-8b0c-289964eb88ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:57.346665 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e60ad173-9503-4471-9d30-33268d468f67 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '42941d5f4dae45c6b1cfa5a803a943d6', 'request_id': 'req-e60ad173-9503-4471-9d30-33268d468f67'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:57.358882 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e60ad173-9503-4471-9d30-33268d468f67 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '42941d5f4dae45c6b1cfa5a803a943d6', 'request_id': 'req-e60ad173-9503-4471-9d30-33268d468f67'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:57.375456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e60ad173-9503-4471-9d30-33268d468f67 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '42941d5f4dae45c6b1cfa5a803a943d6', 'request_id': 'req-e60ad173-9503-4471-9d30-33268d468f67'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:57.410589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e60ad173-9503-4471-9d30-33268d468f67 admin admin] The token cache is being invalidate because project 42941d5f4dae45c6b1cfa5a803a943d6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:57:57.410936 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e60ad173-9503-4471-9d30-33268d468f67 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:57.413663 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e60ad173-9503-4471-9d30-33268d468f67 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:57.426925 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 763/1485] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:57:56 2022] DELETE /identity/v3/projects/42941d5f4dae45c6b1cfa5a803a943d6 => generated 0 bytes in 478 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:57.436671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60ad173-9503-4471-9d30-33268d468f67 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:57.442445 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-e60ad173-9503-4471-9d30-33268d468f67 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:57.470057 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90e30c2-8c18-40c4-8b0c-289964eb88ae None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:57.576638 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59293adb-cf41-48f7-ae8d-24828dc204a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:57.577377 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59293adb-cf41-48f7-ae8d-24828dc204a3 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:57.577665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59293adb-cf41-48f7-ae8d-24828dc204a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:57.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59293adb-cf41-48f7-ae8d-24828dc204a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:57.579883 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-59293adb-cf41-48f7-ae8d-24828dc204a3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:57.592337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60ad173-9503-4471-9d30-33268d468f67 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:57.599103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59293adb-cf41-48f7-ae8d-24828dc204a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:57.600952 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59293adb-cf41-48f7-ae8d-24828dc204a3 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:57.607656 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dd4d30-7ffa-4ef2-b08a-6c05021ef5d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:57.608784 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dd4d30-7ffa-4ef2-b08a-6c05021ef5d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:57.609777 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dd4d30-7ffa-4ef2-b08a-6c05021ef5d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:57.612396 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dd4d30-7ffa-4ef2-b08a-6c05021ef5d3 admin admin] PATH_INFO: `/v3/users/2db3772fb1ee470598f430a41f952937` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:57.614288 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dd4d30-7ffa-4ef2-b08a-6c05021ef5d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2db3772fb1ee470598f430a41f952937)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:57.619961 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 723/1486] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:57.620254 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dd4d30-7ffa-4ef2-b08a-6c05021ef5d3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:57.649423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:57.730481 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-83dd4d30-7ffa-4ef2-b08a-6c05021ef5d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2db3772fb1ee470598f430a41f952937'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:57.731678 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-83dd4d30-7ffa-4ef2-b08a-6c05021ef5d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2db3772fb1ee470598f430a41f952937'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:57.744105 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-83dd4d30-7ffa-4ef2-b08a-6c05021ef5d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2db3772fb1ee470598f430a41f952937'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:57.795416 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-83dd4d30-7ffa-4ef2-b08a-6c05021ef5d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2db3772fb1ee470598f430a41f952937'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:57.818886 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-83dd4d30-7ffa-4ef2-b08a-6c05021ef5d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2db3772fb1ee470598f430a41f952937'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:57.849839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 764/1487] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:57:57 2022] DELETE /identity/v3/users/2db3772fb1ee470598f430a41f952937 => generated 0 bytes in 411 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:58.025834 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a79b7ab-c7b5-42a7-8e81-2a2187387b33 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:58.029203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a79b7ab-c7b5-42a7-8e81-2a2187387b33 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:58.030900 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a79b7ab-c7b5-42a7-8e81-2a2187387b33 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:58.034842 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 724/1488] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 16 00:57:58 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:57:58.075030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc413c5-3ac2-4c21-b6ec-4cc64b949ac4 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:58.075690 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc413c5-3ac2-4c21-b6ec-4cc64b949ac4 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:58.076522 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc413c5-3ac2-4c21-b6ec-4cc64b949ac4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:58.174967 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-6cc413c5-3ac2-4c21-b6ec-4cc64b949ac4 None None] MFA Rules not processed for user `4eba26813e154595a12a67fe4be5eeff`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:58.189633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6cc413c5-3ac2-4c21-b6ec-4cc64b949ac4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:58.310178 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 765/1489] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 16 00:57:58 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:58.489662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a79b7ab-c7b5-42a7-8e81-2a2187387b33 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:58.492210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6a79b7ab-c7b5-42a7-8e81-2a2187387b33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:58.524261 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc413c5-3ac2-4c21-b6ec-4cc64b949ac4 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:58.533140 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6cc413c5-3ac2-4c21-b6ec-4cc64b949ac4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:58.536353 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a79b7ab-c7b5-42a7-8e81-2a2187387b33 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:58.548871 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba311578-b185-4940-a23d-533ba2a4f5c6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:58.550642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba311578-b185-4940-a23d-533ba2a4f5c6 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:58.551266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba311578-b185-4940-a23d-533ba2a4f5c6 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:58.551737 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba311578-b185-4940-a23d-533ba2a4f5c6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:58.565004 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba311578-b185-4940-a23d-533ba2a4f5c6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:58.567963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba311578-b185-4940-a23d-533ba2a4f5c6 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:58.608737 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 725/1490] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:57:58 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:57:58.655160 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc413c5-3ac2-4c21-b6ec-4cc64b949ac4 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:58.660578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2890d5-5b63-4696-a8a1-3202cec74267 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:58.661093 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2890d5-5b63-4696-a8a1-3202cec74267 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:58.661493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2890d5-5b63-4696-a8a1-3202cec74267 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:58.662207 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2890d5-5b63-4696-a8a1-3202cec74267 admin admin] PATH_INFO: `/v3/projects/7d45f968521b40bf8807ee060da82f1e` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:58.663518 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2890d5-5b63-4696-a8a1-3202cec74267 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7d45f968521b40bf8807ee060da82f1e)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:58.665617 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2890d5-5b63-4696-a8a1-3202cec74267 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:58.721291 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:58.744003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-7a2890d5-5b63-4696-a8a1-3202cec74267 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7d45f968521b40bf8807ee060da82f1e', 'request_id': 'req-7a2890d5-5b63-4696-a8a1-3202cec74267'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:58.754853 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-7a2890d5-5b63-4696-a8a1-3202cec74267 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7d45f968521b40bf8807ee060da82f1e', 'request_id': 'req-7a2890d5-5b63-4696-a8a1-3202cec74267'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:58.774798 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-7a2890d5-5b63-4696-a8a1-3202cec74267 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7d45f968521b40bf8807ee060da82f1e', 'request_id': 'req-7a2890d5-5b63-4696-a8a1-3202cec74267'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:58.795098 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-7a2890d5-5b63-4696-a8a1-3202cec74267 admin admin] The token cache is being invalidate because project 7d45f968521b40bf8807ee060da82f1e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:57:58.795406 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-7a2890d5-5b63-4696-a8a1-3202cec74267 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:58.796147 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-7a2890d5-5b63-4696-a8a1-3202cec74267 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:58.799558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 766/1491] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:57:58 2022] DELETE /identity/v3/projects/7d45f968521b40bf8807ee060da82f1e => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:57:58.804958 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac18b571-16cc-4be2-8ee2-f86d87c1fbb0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:58.805227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac18b571-16cc-4be2-8ee2-f86d87c1fbb0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:58.805510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac18b571-16cc-4be2-8ee2-f86d87c1fbb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:58.822394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2890d5-5b63-4696-a8a1-3202cec74267 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:58.824609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-7a2890d5-5b63-4696-a8a1-3202cec74267 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:58.831754 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ac18b571-16cc-4be2-8ee2-f86d87c1fbb0 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:58.833567 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ac18b571-16cc-4be2-8ee2-f86d87c1fbb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:58.923673 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:58.971047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2890d5-5b63-4696-a8a1-3202cec74267 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:58.976526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:58.977178 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:58.977542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:58.977791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a admin admin] PATH_INFO: `/v3/projects/0d2920f753864a4c807705e2a33d0779` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:58.979035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a admin admin] RBAC: Authorizing `identity:delete_project(project_id=0d2920f753864a4c807705e2a33d0779)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:58.984837 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:58.993976 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 726/1492] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:58.994880 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac18b571-16cc-4be2-8ee2-f86d87c1fbb0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:58.996348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ac18b571-16cc-4be2-8ee2-f86d87c1fbb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:59.010812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac18b571-16cc-4be2-8ee2-f86d87c1fbb0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:59.028814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8211206-48e8-4995-a7ac-afedf248b7db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:59.031566 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8211206-48e8-4995-a7ac-afedf248b7db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:59.039455 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8211206-48e8-4995-a7ac-afedf248b7db admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:59.039984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8211206-48e8-4995-a7ac-afedf248b7db admin admin] PATH_INFO: `/v3/projects/8b9dcccdabca4bfba036e822ca020eff` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:59.042835 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8211206-48e8-4995-a7ac-afedf248b7db admin admin] RBAC: Authorizing `identity:delete_project(project_id=8b9dcccdabca4bfba036e822ca020eff)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:59.047289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8211206-48e8-4995-a7ac-afedf248b7db admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:59.087081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0d2920f753864a4c807705e2a33d0779', 'request_id': 'req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.103373 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0d2920f753864a4c807705e2a33d0779', 'request_id': 'req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.114255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0d2920f753864a4c807705e2a33d0779', 'request_id': 'req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.123948 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f8211206-48e8-4995-a7ac-afedf248b7db admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8b9dcccdabca4bfba036e822ca020eff', 'request_id': 'req-f8211206-48e8-4995-a7ac-afedf248b7db'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.130024 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a admin admin] The token cache is being invalidate because project 0d2920f753864a4c807705e2a33d0779 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:57:59.131067 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.132616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.134753 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 767/1493] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:57:58 2022] DELETE /identity/v3/projects/0d2920f753864a4c807705e2a33d0779 => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:57:59.135845 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:59.138217 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:59.139252 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f8211206-48e8-4995-a7ac-afedf248b7db admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8b9dcccdabca4bfba036e822ca020eff', 'request_id': 'req-f8211206-48e8-4995-a7ac-afedf248b7db'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.147256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f8211206-48e8-4995-a7ac-afedf248b7db admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8b9dcccdabca4bfba036e822ca020eff', 'request_id': 'req-f8211206-48e8-4995-a7ac-afedf248b7db'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.162802 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f8211206-48e8-4995-a7ac-afedf248b7db admin admin] The token cache is being invalidate because project 8b9dcccdabca4bfba036e822ca020eff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:57:59.163118 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f8211206-48e8-4995-a7ac-afedf248b7db admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.163947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-f8211206-48e8-4995-a7ac-afedf248b7db admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.165970 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 727/1494] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:57:58 2022] DELETE /identity/v3/projects/8b9dcccdabca4bfba036e822ca020eff => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:59.167436 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c491646d-32ce-40b9-a12f-41abbb77d39a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:59.167624 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c491646d-32ce-40b9-a12f-41abbb77d39a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:59.167897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c491646d-32ce-40b9-a12f-41abbb77d39a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:59.206498 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c491646d-32ce-40b9-a12f-41abbb77d39a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:59.207671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c491646d-32ce-40b9-a12f-41abbb77d39a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:59.228353 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:59.249202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b035ac9-eb75-4440-ab54-e9ee83e3ca3a admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:59.286332 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 728/1495] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:59.290456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c491646d-32ce-40b9-a12f-41abbb77d39a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:59.291851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c491646d-32ce-40b9-a12f-41abbb77d39a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:59.303565 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c491646d-32ce-40b9-a12f-41abbb77d39a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:59.308412 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba11229-c492-4252-8870-fbf9ca677614 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:59.309154 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba11229-c492-4252-8870-fbf9ca677614 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:59.309682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba11229-c492-4252-8870-fbf9ca677614 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:59.309682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba11229-c492-4252-8870-fbf9ca677614 admin admin] PATH_INFO: `/v3/users/4b8fe79c527345e9a8d072b30e9ba349` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:59.310757 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba11229-c492-4252-8870-fbf9ca677614 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b8fe79c527345e9a8d072b30e9ba349)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:59.312720 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba11229-c492-4252-8870-fbf9ca677614 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:59.318095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:59.318747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:59.319071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:59.319357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803 admin admin] PATH_INFO: `/v3/projects/692b78de7ff4450ab6d62431ad807c94` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:59.320491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803 admin admin] RBAC: Authorizing `identity:delete_project(project_id=692b78de7ff4450ab6d62431ad807c94)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:59.322432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:59.362470 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8ba11229-c492-4252-8870-fbf9ca677614 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b8fe79c527345e9a8d072b30e9ba349'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.363217 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8ba11229-c492-4252-8870-fbf9ca677614 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b8fe79c527345e9a8d072b30e9ba349'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.371038 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8ba11229-c492-4252-8870-fbf9ca677614 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b8fe79c527345e9a8d072b30e9ba349'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.372311 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8ba11229-c492-4252-8870-fbf9ca677614 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b8fe79c527345e9a8d072b30e9ba349'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.375582 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '692b78de7ff4450ab6d62431ad807c94', 'request_id': 'req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.377102 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:59.384258 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '692b78de7ff4450ab6d62431ad807c94', 'request_id': 'req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.388844 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8ba11229-c492-4252-8870-fbf9ca677614 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b8fe79c527345e9a8d072b30e9ba349'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.392245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '692b78de7ff4450ab6d62431ad807c94', 'request_id': 'req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.402526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 729/1496] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:57:59 2022] DELETE /identity/v3/users/4b8fe79c527345e9a8d072b30e9ba349 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:59.403830 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325968de-5993-4a63-8c43-935d37fc7d32 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:59.404150 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325968de-5993-4a63-8c43-935d37fc7d32 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:59.404390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325968de-5993-4a63-8c43-935d37fc7d32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:59.408399 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803 admin admin] The token cache is being invalidate because project 692b78de7ff4450ab6d62431ad807c94 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:57:59.408854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.409617 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:57:59.414528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 768/1497] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:57:59 2022] DELETE /identity/v3/projects/692b78de7ff4450ab6d62431ad807c94 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:57:59.414528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:59.415114 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:59.427061 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-325968de-5993-4a63-8c43-935d37fc7d32 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:59.428542 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-325968de-5993-4a63-8c43-935d37fc7d32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:59.523187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 730/1498] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:59.525405 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aef65b7-d3e7-440a-b9d2-c690b89567db None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:59.525692 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aef65b7-d3e7-440a-b9d2-c690b89567db None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:59.526531 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aef65b7-d3e7-440a-b9d2-c690b89567db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:59.552686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-6aef65b7-d3e7-440a-b9d2-c690b89567db None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:59.554157 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6aef65b7-d3e7-440a-b9d2-c690b89567db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:59.565233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 731/1499] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:59.570231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ced7bb3-14b7-4cb9-b6d4-c7a999221803 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:59.576031 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f125204b-faeb-4af1-a9da-02f6750f1c02 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:59.576569 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f125204b-faeb-4af1-a9da-02f6750f1c02 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:59.577377 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f125204b-faeb-4af1-a9da-02f6750f1c02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:59.586465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d565b5-098f-4922-ae04-bf07a8bec63c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:59.587531 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d565b5-098f-4922-ae04-bf07a8bec63c admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:59.588262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d565b5-098f-4922-ae04-bf07a8bec63c admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:59.588740 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d565b5-098f-4922-ae04-bf07a8bec63c admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:59.591345 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d565b5-098f-4922-ae04-bf07a8bec63c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:57:59.592149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d565b5-098f-4922-ae04-bf07a8bec63c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:59.596652 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d565b5-098f-4922-ae04-bf07a8bec63c admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:59.599749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f125204b-faeb-4af1-a9da-02f6750f1c02 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:59.601603 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f125204b-faeb-4af1-a9da-02f6750f1c02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:59.612585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 732/1500] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:59.614292 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ff11fe-af9d-4525-a39b-946570f2424c None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:59.614509 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 769/1501] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:57:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:57:59.614612 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ff11fe-af9d-4525-a39b-946570f2424c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:59.615421 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ff11fe-af9d-4525-a39b-946570f2424c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:59.619109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d565b5-098f-4922-ae04-bf07a8bec63c admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:59.634349 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d565b5-098f-4922-ae04-bf07a8bec63c admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:59.642124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07e4f9e-608f-4b1b-a545-598287800abe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:59.642835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07e4f9e-608f-4b1b-a545-598287800abe admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:59.643230 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07e4f9e-608f-4b1b-a545-598287800abe admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:59.643603 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07e4f9e-608f-4b1b-a545-598287800abe admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:59.644544 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07e4f9e-608f-4b1b-a545-598287800abe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:57:59.644921 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07e4f9e-608f-4b1b-a545-598287800abe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:59.646612 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07e4f9e-608f-4b1b-a545-598287800abe admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:59.647246 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-53ff11fe-af9d-4525-a39b-946570f2424c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:57:59.648697 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-53ff11fe-af9d-4525-a39b-946570f2424c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:57:59.657972 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 733/1502] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:57:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:57:59.662538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ff11fe-af9d-4525-a39b-946570f2424c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:59.675072 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ff11fe-af9d-4525-a39b-946570f2424c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:59.679592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 770/1503] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:57:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:57:59.682665 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07e4f9e-608f-4b1b-a545-598287800abe admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:59.684031 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472a9810-80b7-4c50-bfea-7f22e0561fe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:59.684968 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472a9810-80b7-4c50-bfea-7f22e0561fe3 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:59.685482 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472a9810-80b7-4c50-bfea-7f22e0561fe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:59.686138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472a9810-80b7-4c50-bfea-7f22e0561fe3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:59.687116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472a9810-80b7-4c50-bfea-7f22e0561fe3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:59.689692 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472a9810-80b7-4c50-bfea-7f22e0561fe3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:59.707110 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07e4f9e-608f-4b1b-a545-598287800abe admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:59.724326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a341c3-ed3e-4d49-a357-bd2420665470 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:59.724969 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a341c3-ed3e-4d49-a357-bd2420665470 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:59.725251 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a341c3-ed3e-4d49-a357-bd2420665470 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:59.725564 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a341c3-ed3e-4d49-a357-bd2420665470 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:59.726266 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a341c3-ed3e-4d49-a357-bd2420665470 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:57:59.726547 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a341c3-ed3e-4d49-a357-bd2420665470 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:59.735156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a341c3-ed3e-4d49-a357-bd2420665470 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:59.736149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 734/1504] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:59 2022] POST /identity/v3/projects => generated 389 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:59.737600 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472a9810-80b7-4c50-bfea-7f22e0561fe3 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:59.740874 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:57:59.779078 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 771/1505] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:57:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:57:59.780056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a341c3-ed3e-4d49-a357-bd2420665470 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:59.806425 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a341c3-ed3e-4d49-a357-bd2420665470 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:59.866243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472a9810-80b7-4c50-bfea-7f22e0561fe3 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:57:59.878688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6770a7e5-e821-4c74-a2c9-633cbf411eb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:59.880639 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6770a7e5-e821-4c74-a2c9-633cbf411eb6 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:59.881254 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6770a7e5-e821-4c74-a2c9-633cbf411eb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:59.882347 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6770a7e5-e821-4c74-a2c9-633cbf411eb6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:59.885202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6770a7e5-e821-4c74-a2c9-633cbf411eb6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:59.892243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6770a7e5-e821-4c74-a2c9-633cbf411eb6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:59.958800 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b824b2-20a0-44dc-a681-39d9dc645ab6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:57:59.959017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 735/1506] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:59 2022] POST /identity/v3/projects => generated 387 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:57:59.960089 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6770a7e5-e821-4c74-a2c9-633cbf411eb6 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:57:59.963126 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b824b2-20a0-44dc-a681-39d9dc645ab6 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:57:59.963495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b824b2-20a0-44dc-a681-39d9dc645ab6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:57:59.963797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b824b2-20a0-44dc-a681-39d9dc645ab6 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:57:59.964603 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b824b2-20a0-44dc-a681-39d9dc645ab6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:57:59.966775 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b824b2-20a0-44dc-a681-39d9dc645ab6 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:57:59.999020 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 772/1507] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:59 2022] POST /identity/v3/users => generated 295 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:00.001859 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b824b2-20a0-44dc-a681-39d9dc645ab6 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:00.002888 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:DELETE=2,keystone:SELECT=96,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:00.017072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b824b2-20a0-44dc-a681-39d9dc645ab6 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:00.075723 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6770a7e5-e821-4c74-a2c9-633cbf411eb6 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:00.082774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c732fc47-7b01-488d-acff-53d4664457bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:00.083655 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c732fc47-7b01-488d-acff-53d4664457bb admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:00.084102 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c732fc47-7b01-488d-acff-53d4664457bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:00.084520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c732fc47-7b01-488d-acff-53d4664457bb admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:00.085722 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c732fc47-7b01-488d-acff-53d4664457bb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:00.088962 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c732fc47-7b01-488d-acff-53d4664457bb admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:00.133489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 736/1508] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:57:59 2022] POST /identity/v3/projects => generated 385 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:00.134450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c732fc47-7b01-488d-acff-53d4664457bb admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:00.189176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b397f7-105e-411d-ab9c-4228176ede53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:00.190300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b397f7-105e-411d-ab9c-4228176ede53 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:00.190300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b397f7-105e-411d-ab9c-4228176ede53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:00.190525 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b397f7-105e-411d-ab9c-4228176ede53 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:00.191180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b397f7-105e-411d-ab9c-4228176ede53 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:00.193165 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b397f7-105e-411d-ab9c-4228176ede53 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:00.228691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c732fc47-7b01-488d-acff-53d4664457bb admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:00.229654 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 773/1509] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:57:59 2022] POST /identity/v3/users => generated 294 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:00.231884 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b397f7-105e-411d-ab9c-4228176ede53 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:00.240050 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ac64b1-3c0f-4473-825b-b9002b641c29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:00.240691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ac64b1-3c0f-4473-825b-b9002b641c29 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:00.240968 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ac64b1-3c0f-4473-825b-b9002b641c29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:00.241235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ac64b1-3c0f-4473-825b-b9002b641c29 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:00.242070 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ac64b1-3c0f-4473-825b-b9002b641c29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:00.243408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b397f7-105e-411d-ab9c-4228176ede53 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:00.243726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ac64b1-3c0f-4473-825b-b9002b641c29 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:00.252749 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94210b45-7d77-410b-bae1-9a094316672f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:00.253496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94210b45-7d77-410b-bae1-9a094316672f admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:00.253972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94210b45-7d77-410b-bae1-9a094316672f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:00.254328 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94210b45-7d77-410b-bae1-9a094316672f admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:00.256122 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94210b45-7d77-410b-bae1-9a094316672f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:00.258393 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94210b45-7d77-410b-bae1-9a094316672f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:00.259230 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 737/1510] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:00 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:00.260977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ac64b1-3c0f-4473-825b-b9002b641c29 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:00.273096 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ac64b1-3c0f-4473-825b-b9002b641c29 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:00.281931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5293e98-5086-434e-99dc-2dd6b3f01380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:00.283026 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5293e98-5086-434e-99dc-2dd6b3f01380 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:00.283381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5293e98-5086-434e-99dc-2dd6b3f01380 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:00.283878 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5293e98-5086-434e-99dc-2dd6b3f01380 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:00.285050 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5293e98-5086-434e-99dc-2dd6b3f01380 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:00.287807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5293e98-5086-434e-99dc-2dd6b3f01380 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:00.294836 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 774/1511] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:00 2022] POST /identity/v3/users => generated 293 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:00.296205 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94210b45-7d77-410b-bae1-9a094316672f admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:00.307279 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 738/1512] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:00 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:00.308336 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5293e98-5086-434e-99dc-2dd6b3f01380 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:00.311991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94210b45-7d77-410b-bae1-9a094316672f admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:00.318243 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e767cbf-6048-4bda-93b4-aacb3bde33b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:00.318802 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e767cbf-6048-4bda-93b4-aacb3bde33b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:00.319123 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e767cbf-6048-4bda-93b4-aacb3bde33b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:00.319536 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e767cbf-6048-4bda-93b4-aacb3bde33b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:00.320221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e767cbf-6048-4bda-93b4-aacb3bde33b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:00.322260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e767cbf-6048-4bda-93b4-aacb3bde33b4 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:00.323154 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5293e98-5086-434e-99dc-2dd6b3f01380 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:00.330270 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a476493-4bb6-41ed-98fa-6799b3bf01f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:00.331448 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a476493-4bb6-41ed-98fa-6799b3bf01f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:00.332360 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a476493-4bb6-41ed-98fa-6799b3bf01f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:00.332644 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a476493-4bb6-41ed-98fa-6799b3bf01f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:00.333315 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a476493-4bb6-41ed-98fa-6799b3bf01f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:00.335350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a476493-4bb6-41ed-98fa-6799b3bf01f5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:00.341255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 775/1513] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:00 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:00.341255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e767cbf-6048-4bda-93b4-aacb3bde33b4 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:00.352008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 739/1514] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:00 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:00.352008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a476493-4bb6-41ed-98fa-6799b3bf01f5 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:00.353094 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e767cbf-6048-4bda-93b4-aacb3bde33b4 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:00.364023 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a476493-4bb6-41ed-98fa-6799b3bf01f5 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:00.370819 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5040e4-b0ba-45d0-8508-f8bae75b7247 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:00.371881 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5040e4-b0ba-45d0-8508-f8bae75b7247 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:00.372309 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5040e4-b0ba-45d0-8508-f8bae75b7247 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:00.372594 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5040e4-b0ba-45d0-8508-f8bae75b7247 admin admin] PATH_INFO: `/v3/projects/56dafe1503724a8387b09e7ced1f22b6/users/71b22c2c96994a5a89749a7cb7753b66/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:00.373465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb5c20b-e38b-4c0d-ba01-2e5d0f7d3fa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:00.374526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb5c20b-e38b-4c0d-ba01-2e5d0f7d3fa3 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:00.374819 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb5c20b-e38b-4c0d-ba01-2e5d0f7d3fa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:00.375155 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb5c20b-e38b-4c0d-ba01-2e5d0f7d3fa3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:00.378461 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb5c20b-e38b-4c0d-ba01-2e5d0f7d3fa3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:00.380324 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb5c20b-e38b-4c0d-ba01-2e5d0f7d3fa3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:00.398609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 776/1515] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:00 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:00.400277 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb5c20b-e38b-4c0d-ba01-2e5d0f7d3fa3 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:00.404393 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5040e4-b0ba-45d0-8508-f8bae75b7247 admin admin] RBAC: Authorizing `identity:create_grant(project_id=56dafe1503724a8387b09e7ced1f22b6, user_id=71b22c2c96994a5a89749a7cb7753b66, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:00.410322 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5040e4-b0ba-45d0-8508-f8bae75b7247 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:00.420928 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb5c20b-e38b-4c0d-ba01-2e5d0f7d3fa3 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:00.445351 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598d7e51-6ed2-4b0a-b1a6-9a1871225a18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:00.445351 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598d7e51-6ed2-4b0a-b1a6-9a1871225a18 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:00.445351 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598d7e51-6ed2-4b0a-b1a6-9a1871225a18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:00.445351 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598d7e51-6ed2-4b0a-b1a6-9a1871225a18 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:00.446387 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598d7e51-6ed2-4b0a-b1a6-9a1871225a18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:00.446387 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598d7e51-6ed2-4b0a-b1a6-9a1871225a18 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:00.447039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 740/1516] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:00 2022] PUT /identity/v3/projects/56dafe1503724a8387b09e7ced1f22b6/users/71b22c2c96994a5a89749a7cb7753b66/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:00.448254 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5040e4-b0ba-45d0-8508-f8bae75b7247 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:00.455705 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 777/1517] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:00 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:00.457009 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18848af3-05e1-451b-9545-2d088298450d None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:00.457434 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18848af3-05e1-451b-9545-2d088298450d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:00.457639 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18848af3-05e1-451b-9545-2d088298450d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:00.504690 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-18848af3-05e1-451b-9545-2d088298450d None None] MFA Rules not processed for user `71b22c2c96994a5a89749a7cb7753b66`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:00.506479 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-18848af3-05e1-451b-9545-2d088298450d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:00.532954 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=91,keystone:INSERT=9 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:00.566228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:00.573155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5040e4-b0ba-45d0-8508-f8bae75b7247 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:00.580522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89f7bc3-ba2f-4655-aa2b-67c857809daa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:00.581613 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f7bc3-ba2f-4655-aa2b-67c857809daa admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:00.581897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f7bc3-ba2f-4655-aa2b-67c857809daa admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:00.582296 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f7bc3-ba2f-4655-aa2b-67c857809daa admin admin] PATH_INFO: `/v3/projects/35ea763c74bf4281a2492556062f5b78/users/d1df414c5cf94f2c8ee44ad1b01b6ac6/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:00.603258 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89f7bc3-ba2f-4655-aa2b-67c857809daa admin admin] RBAC: Authorizing `identity:create_grant(project_id=35ea763c74bf4281a2492556062f5b78, user_id=d1df414c5cf94f2c8ee44ad1b01b6ac6, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:00.607369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89f7bc3-ba2f-4655-aa2b-67c857809daa admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:00.626221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 778/1518] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:00 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:00.626221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18848af3-05e1-451b-9545-2d088298450d None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:00.636305 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 741/1519] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:00 2022] PUT /identity/v3/projects/35ea763c74bf4281a2492556062f5b78/users/d1df414c5cf94f2c8ee44ad1b01b6ac6/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:00.637198 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18848af3-05e1-451b-9545-2d088298450d None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:00.637968 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89f7bc3-ba2f-4655-aa2b-67c857809daa admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:00.725281 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9123b4-4c4e-48db-95f6-d4aa30208635 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:00.726236 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9123b4-4c4e-48db-95f6-d4aa30208635 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:00.727851 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9123b4-4c4e-48db-95f6-d4aa30208635 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:00.727851 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9123b4-4c4e-48db-95f6-d4aa30208635 admin admin] PATH_INFO: `/v3/projects/8d819a5b4ad146bb8c3eccf8c8e2d7e9/users/b484e5d8140d4bc2894c663466f47c61/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:00.754740 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9123b4-4c4e-48db-95f6-d4aa30208635 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d819a5b4ad146bb8c3eccf8c8e2d7e9, user_id=b484e5d8140d4bc2894c663466f47c61, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:00.761261 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9123b4-4c4e-48db-95f6-d4aa30208635 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:00.779810 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89f7bc3-ba2f-4655-aa2b-67c857809daa admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:00.787950 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 779/1520] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:00 2022] PUT /identity/v3/projects/8d819a5b4ad146bb8c3eccf8c8e2d7e9/users/b484e5d8140d4bc2894c663466f47c61/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:00.790824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a96201-134c-4700-b6f3-9c8512558b49 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:00.791152 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a96201-134c-4700-b6f3-9c8512558b49 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:00.791416 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a96201-134c-4700-b6f3-9c8512558b49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:00.869574 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-b6a96201-134c-4700-b6f3-9c8512558b49 None None] MFA Rules not processed for user `d1df414c5cf94f2c8ee44ad1b01b6ac6`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:00.870830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b6a96201-134c-4700-b6f3-9c8512558b49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:00.912424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2877b72d-66c7-4d05-aeaa-bec04b630ba5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:00.913159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2877b72d-66c7-4d05-aeaa-bec04b630ba5 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:00.913357 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2877b72d-66c7-4d05-aeaa-bec04b630ba5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:00.913634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2877b72d-66c7-4d05-aeaa-bec04b630ba5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:00.914346 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2877b72d-66c7-4d05-aeaa-bec04b630ba5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:00.916260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2877b72d-66c7-4d05-aeaa-bec04b630ba5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:00.967507 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 742/1521] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:00 2022] POST /identity/v3/projects => generated 387 bytes in 330 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:00.971316 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c6ebc3-18f7-4ac5-b91f-0c56c9cc4878 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:00.972036 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 780/1522] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:00 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:00.972488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c6ebc3-18f7-4ac5-b91f-0c56c9cc4878 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:00.973224 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a96201-134c-4700-b6f3-9c8512558b49 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:00.973549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c6ebc3-18f7-4ac5-b91f-0c56c9cc4878 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:00.975266 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b6a96201-134c-4700-b6f3-9c8512558b49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:01.000050 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a96201-134c-4700-b6f3-9c8512558b49 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:01.012656 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-20c6ebc3-18f7-4ac5-b91f-0c56c9cc4878 None None] MFA Rules not processed for user `b484e5d8140d4bc2894c663466f47c61`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:01.014335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-20c6ebc3-18f7-4ac5-b91f-0c56c9cc4878 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:01.035587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:01.106183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5f472f-dde0-49a2-866e-01d30c209235 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:01.106769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5f472f-dde0-49a2-866e-01d30c209235 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:01.107100 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5f472f-dde0-49a2-866e-01d30c209235 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:01.107378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5f472f-dde0-49a2-866e-01d30c209235 admin admin] PATH_INFO: `/v3/projects/8540cf30cae44346aef35f64936213df` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:01.108564 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5f472f-dde0-49a2-866e-01d30c209235 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8540cf30cae44346aef35f64936213df)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:01.110784 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d5f472f-dde0-49a2-866e-01d30c209235 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:01.115433 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:01.125284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 743/1523] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:00 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:01.126246 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c6ebc3-18f7-4ac5-b91f-0c56c9cc4878 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:01.142679 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c6ebc3-18f7-4ac5-b91f-0c56c9cc4878 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:01.147931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926902d0-2fd4-4912-897b-3a35118d7d87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:01.148886 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926902d0-2fd4-4912-897b-3a35118d7d87 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:01.149068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926902d0-2fd4-4912-897b-3a35118d7d87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:01.149477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926902d0-2fd4-4912-897b-3a35118d7d87 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:01.150370 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926902d0-2fd4-4912-897b-3a35118d7d87 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:01.152629 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926902d0-2fd4-4912-897b-3a35118d7d87 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:01.181173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4d5f472f-dde0-49a2-866e-01d30c209235 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8540cf30cae44346aef35f64936213df', 'request_id': 'req-4d5f472f-dde0-49a2-866e-01d30c209235'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:01.191399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 744/1524] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:01 2022] POST /identity/v3/users => generated 300 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:01.192580 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926902d0-2fd4-4912-897b-3a35118d7d87 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:01.198662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4d5f472f-dde0-49a2-866e-01d30c209235 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8540cf30cae44346aef35f64936213df', 'request_id': 'req-4d5f472f-dde0-49a2-866e-01d30c209235'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:01.207931 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4d5f472f-dde0-49a2-866e-01d30c209235 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8540cf30cae44346aef35f64936213df', 'request_id': 'req-4d5f472f-dde0-49a2-866e-01d30c209235'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:01.221373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926902d0-2fd4-4912-897b-3a35118d7d87 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:01.228130 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4d5f472f-dde0-49a2-866e-01d30c209235 admin admin] The token cache is being invalidate because project 8540cf30cae44346aef35f64936213df was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:01.228516 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4d5f472f-dde0-49a2-866e-01d30c209235 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:01.229282 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4d5f472f-dde0-49a2-866e-01d30c209235 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:01.231542 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 781/1525] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:00 2022] DELETE /identity/v3/projects/8540cf30cae44346aef35f64936213df => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:01.232488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5f472f-dde0-49a2-866e-01d30c209235 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:01.235452 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4d5f472f-dde0-49a2-866e-01d30c209235 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:01.315719 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67144fd6-5608-4c60-ada1-5c21faf39f7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:01.316343 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67144fd6-5608-4c60-ada1-5c21faf39f7e admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:01.316633 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67144fd6-5608-4c60-ada1-5c21faf39f7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:01.316908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67144fd6-5608-4c60-ada1-5c21faf39f7e admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:01.317550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67144fd6-5608-4c60-ada1-5c21faf39f7e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:01.319944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67144fd6-5608-4c60-ada1-5c21faf39f7e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:01.326782 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5f472f-dde0-49a2-866e-01d30c209235 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:01.331324 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bb186a-ba56-4482-8059-b54f6bd804e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:01.332819 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bb186a-ba56-4482-8059-b54f6bd804e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:01.332819 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bb186a-ba56-4482-8059-b54f6bd804e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:01.333110 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bb186a-ba56-4482-8059-b54f6bd804e8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:01.333762 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bb186a-ba56-4482-8059-b54f6bd804e8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:01.335773 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bb186a-ba56-4482-8059-b54f6bd804e8 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:01.361563 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 745/1526] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:01 2022] POST /identity/v3/projects => generated 385 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:01.361563 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67144fd6-5608-4c60-ada1-5c21faf39f7e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:01.362720 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-67144fd6-5608-4c60-ada1-5c21faf39f7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:01.375191 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 782/1527] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:01 2022] POST /identity/v3/projects => generated 387 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:01.376971 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4916d90a-a0fe-4498-8779-d6990d856bbf None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:01.377146 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4916d90a-a0fe-4498-8779-d6990d856bbf None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:01.377363 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4916d90a-a0fe-4498-8779-d6990d856bbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:01.414748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-4916d90a-a0fe-4498-8779-d6990d856bbf None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:01.417645 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4916d90a-a0fe-4498-8779-d6990d856bbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:01.459916 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:01.492503 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67144fd6-5608-4c60-ada1-5c21faf39f7e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:01.518332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 783/1528] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:01.520191 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4916d90a-a0fe-4498-8779-d6990d856bbf None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:01.521633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4916d90a-a0fe-4498-8779-d6990d856bbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:01.535541 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4916d90a-a0fe-4498-8779-d6990d856bbf None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:01.542678 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0433e7d3-ab75-4cb0-8dea-af49eb7774ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:01.543349 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0433e7d3-ab75-4cb0-8dea-af49eb7774ce admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:01.543631 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0433e7d3-ab75-4cb0-8dea-af49eb7774ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:01.543901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0433e7d3-ab75-4cb0-8dea-af49eb7774ce admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:01.544651 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0433e7d3-ab75-4cb0-8dea-af49eb7774ce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:01.546609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0433e7d3-ab75-4cb0-8dea-af49eb7774ce admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:01.578365 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 784/1529] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:01 2022] POST /identity/v3/users => generated 299 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:01.579972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0433e7d3-ab75-4cb0-8dea-af49eb7774ce admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:01.581461 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77930f74-0b8a-4a47-9ebb-3a1a79931d7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:01.582195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77930f74-0b8a-4a47-9ebb-3a1a79931d7b admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:01.582649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77930f74-0b8a-4a47-9ebb-3a1a79931d7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:01.582960 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77930f74-0b8a-4a47-9ebb-3a1a79931d7b admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:01.583624 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77930f74-0b8a-4a47-9ebb-3a1a79931d7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:01.585488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77930f74-0b8a-4a47-9ebb-3a1a79931d7b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:01.594173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0433e7d3-ab75-4cb0-8dea-af49eb7774ce admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:01.600057 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 746/1530] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:01 2022] GET /identity/v3/roles => generated 1420 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:01.600713 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d69e55-e180-4b61-854b-90d955c89d3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:01.601445 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bf02fa-f73b-4920-8867-84e914269745 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:01.601812 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d69e55-e180-4b61-854b-90d955c89d3e admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:01.601952 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bf02fa-f73b-4920-8867-84e914269745 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:01.602131 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bf02fa-f73b-4920-8867-84e914269745 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:01.602814 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d69e55-e180-4b61-854b-90d955c89d3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:01.603412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d69e55-e180-4b61-854b-90d955c89d3e admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:01.604320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d69e55-e180-4b61-854b-90d955c89d3e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:01.607247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d69e55-e180-4b61-854b-90d955c89d3e admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:01.631244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-03bf02fa-f73b-4920-8867-84e914269745 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:01.632530 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-03bf02fa-f73b-4920-8867-84e914269745 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:01.637580 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 785/1531] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:01 2022] POST /identity/v3/users => generated 300 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:01.637580 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d69e55-e180-4b61-854b-90d955c89d3e admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:01.647025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 747/1532] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:01.648045 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03bf02fa-f73b-4920-8867-84e914269745 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:01.651329 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d69e55-e180-4b61-854b-90d955c89d3e admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:01.654715 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:01.657322 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d124a945-5ab3-4199-aaf7-fc9aace74389 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:01.657927 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d124a945-5ab3-4199-aaf7-fc9aace74389 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:01.658733 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d124a945-5ab3-4199-aaf7-fc9aace74389 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:01.659054 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03bf02fa-f73b-4920-8867-84e914269745 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:01.659186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d124a945-5ab3-4199-aaf7-fc9aace74389 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:01.660087 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d124a945-5ab3-4199-aaf7-fc9aace74389 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:01.663178 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d124a945-5ab3-4199-aaf7-fc9aace74389 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:01.670721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7703164b-fe0c-4617-8d0a-9b193b268359 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:01.672061 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7703164b-fe0c-4617-8d0a-9b193b268359 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:01.672468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7703164b-fe0c-4617-8d0a-9b193b268359 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:01.672819 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7703164b-fe0c-4617-8d0a-9b193b268359 admin admin] PATH_INFO: `/v3/projects/ab3e02221593480a8b183f9cb050dd38/users/a4dd8f6f10c741fbbce49c6975e13256/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:01.681246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 786/1533] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:01 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:01.682841 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d124a945-5ab3-4199-aaf7-fc9aace74389 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:01.693017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7703164b-fe0c-4617-8d0a-9b193b268359 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab3e02221593480a8b183f9cb050dd38, user_id=a4dd8f6f10c741fbbce49c6975e13256, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:01.696320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7703164b-fe0c-4617-8d0a-9b193b268359 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:01.704920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d124a945-5ab3-4199-aaf7-fc9aace74389 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:01.711232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3e3474-443d-452a-8dcf-22fb03d02273 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:01.712135 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3e3474-443d-452a-8dcf-22fb03d02273 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:01.712521 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3e3474-443d-452a-8dcf-22fb03d02273 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:01.712810 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3e3474-443d-452a-8dcf-22fb03d02273 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:01.713467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3e3474-443d-452a-8dcf-22fb03d02273 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:01.715846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3e3474-443d-452a-8dcf-22fb03d02273 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:01.716191 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 748/1534] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:01 2022] PUT /identity/v3/projects/ab3e02221593480a8b183f9cb050dd38/users/a4dd8f6f10c741fbbce49c6975e13256/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:01.717224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7703164b-fe0c-4617-8d0a-9b193b268359 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:01.731361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 787/1535] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:01 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:01.735577 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3e3474-443d-452a-8dcf-22fb03d02273 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:01.746691 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3e3474-443d-452a-8dcf-22fb03d02273 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:01.802677 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7703164b-fe0c-4617-8d0a-9b193b268359 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:01.808054 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356cad54-4715-4251-8bf6-2007d81fa30b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:01.809183 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356cad54-4715-4251-8bf6-2007d81fa30b admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:01.809471 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356cad54-4715-4251-8bf6-2007d81fa30b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:01.809750 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356cad54-4715-4251-8bf6-2007d81fa30b admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:01.810582 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356cad54-4715-4251-8bf6-2007d81fa30b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:58:01.810941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356cad54-4715-4251-8bf6-2007d81fa30b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:01.812573 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356cad54-4715-4251-8bf6-2007d81fa30b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:01.814980 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9f709d-f5bc-48a7-9b8d-c243152d28e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:01.815720 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9f709d-f5bc-48a7-9b8d-c243152d28e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:01.816087 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9f709d-f5bc-48a7-9b8d-c243152d28e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:01.816429 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9f709d-f5bc-48a7-9b8d-c243152d28e7 admin admin] PATH_INFO: `/v3/projects/996d26fd06a943258239da182d539a21/users/e9bee6efcb1c4053a7cc8894a3d289dd/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:01.831136 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 749/1536] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:58:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:58:01.832024 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356cad54-4715-4251-8bf6-2007d81fa30b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:01.836630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9f709d-f5bc-48a7-9b8d-c243152d28e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=996d26fd06a943258239da182d539a21, user_id=e9bee6efcb1c4053a7cc8894a3d289dd, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:01.841884 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9f709d-f5bc-48a7-9b8d-c243152d28e7 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:01.845945 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356cad54-4715-4251-8bf6-2007d81fa30b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:01.851506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66be2ee7-3f37-4425-885e-f0188b3c9fc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:01.852847 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66be2ee7-3f37-4425-885e-f0188b3c9fc0 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:01.853155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66be2ee7-3f37-4425-885e-f0188b3c9fc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:01.853436 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66be2ee7-3f37-4425-885e-f0188b3c9fc0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:01.854366 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66be2ee7-3f37-4425-885e-f0188b3c9fc0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:01.857410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66be2ee7-3f37-4425-885e-f0188b3c9fc0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:01.861374 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 788/1537] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:01 2022] PUT /identity/v3/projects/996d26fd06a943258239da182d539a21/users/e9bee6efcb1c4053a7cc8894a3d289dd/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:01.861374 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9f709d-f5bc-48a7-9b8d-c243152d28e7 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:01.874526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 750/1538] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:01 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:01.875422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66be2ee7-3f37-4425-885e-f0188b3c9fc0 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:01.887866 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66be2ee7-3f37-4425-885e-f0188b3c9fc0 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:01.957604 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9f709d-f5bc-48a7-9b8d-c243152d28e7 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:01.964322 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6e4e13-ba4c-462b-ba5f-fe15ff24cc1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:01.966151 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6e4e13-ba4c-462b-ba5f-fe15ff24cc1b admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:01.966937 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6e4e13-ba4c-462b-ba5f-fe15ff24cc1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:01.967419 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6e4e13-ba4c-462b-ba5f-fe15ff24cc1b admin admin] PATH_INFO: `/v3/projects/f88dc43c613e45aab8456768d988fb0c/users/584927648cab49af9d38ad8d304e5ee1/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:01.981982 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ef303e-74e2-48ce-aef4-b3be2e313c1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:01.982766 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ef303e-74e2-48ce-aef4-b3be2e313c1a admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:01.983093 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ef303e-74e2-48ce-aef4-b3be2e313c1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:01.983439 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ef303e-74e2-48ce-aef4-b3be2e313c1a admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:01.984697 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ef303e-74e2-48ce-aef4-b3be2e313c1a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:01.986290 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:01.989185 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ef303e-74e2-48ce-aef4-b3be2e313c1a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:01.992148 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6e4e13-ba4c-462b-ba5f-fe15ff24cc1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f88dc43c613e45aab8456768d988fb0c, user_id=584927648cab49af9d38ad8d304e5ee1, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:01.995846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6e4e13-ba4c-462b-ba5f-fe15ff24cc1b admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:02.027093 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 789/1539] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:01 2022] PUT /identity/v3/projects/f88dc43c613e45aab8456768d988fb0c/users/584927648cab49af9d38ad8d304e5ee1/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:02.027093 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6e4e13-ba4c-462b-ba5f-fe15ff24cc1b admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:02.035149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 751/1540] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:01 2022] POST /identity/v3/projects => generated 385 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:02.036003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ef303e-74e2-48ce-aef4-b3be2e313c1a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:02.089739 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:02.104913 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ef303e-74e2-48ce-aef4-b3be2e313c1a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:02.106991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6e4e13-ba4c-462b-ba5f-fe15ff24cc1b admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:02.111008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0850fa-464c-4f43-9eee-1b520d8065ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:02.111694 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0850fa-464c-4f43-9eee-1b520d8065ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:02.112081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72d454a-413f-41c4-ab81-f03a97de05a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:02.112222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0850fa-464c-4f43-9eee-1b520d8065ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:02.112404 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0850fa-464c-4f43-9eee-1b520d8065ed admin admin] PATH_INFO: `/v3/projects/ab3e02221593480a8b183f9cb050dd38/users/a4dd8f6f10c741fbbce49c6975e13256/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:02.112760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72d454a-413f-41c4-ab81-f03a97de05a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:02.113101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72d454a-413f-41c4-ab81-f03a97de05a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:02.113435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72d454a-413f-41c4-ab81-f03a97de05a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:02.114140 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72d454a-413f-41c4-ab81-f03a97de05a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:02.114492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0850fa-464c-4f43-9eee-1b520d8065ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab3e02221593480a8b183f9cb050dd38, user_id=a4dd8f6f10c741fbbce49c6975e13256, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:02.116072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72d454a-413f-41c4-ab81-f03a97de05a8 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:02.117326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0850fa-464c-4f43-9eee-1b520d8065ed admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:02.130952 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 790/1541] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:02 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:02.131956 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72d454a-413f-41c4-ab81-f03a97de05a8 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:02.134955 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 752/1542] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:02 2022] PUT /identity/v3/projects/ab3e02221593480a8b183f9cb050dd38/users/a4dd8f6f10c741fbbce49c6975e13256/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:02.136265 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0850fa-464c-4f43-9eee-1b520d8065ed admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:02.155407 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72d454a-413f-41c4-ab81-f03a97de05a8 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:02.236781 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0850fa-464c-4f43-9eee-1b520d8065ed admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:02.244693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df7dcad-f4fa-4757-9397-5709bd1c098e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:02.245707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df7dcad-f4fa-4757-9397-5709bd1c098e admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:02.246706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df7dcad-f4fa-4757-9397-5709bd1c098e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:02.248100 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df7dcad-f4fa-4757-9397-5709bd1c098e admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:02.248989 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df7dcad-f4fa-4757-9397-5709bd1c098e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:02.251548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df7dcad-f4fa-4757-9397-5709bd1c098e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:02.288341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 753/1543] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:02 2022] POST /identity/v3/users => generated 293 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:02.289196 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df7dcad-f4fa-4757-9397-5709bd1c098e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:02.301468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df7dcad-f4fa-4757-9397-5709bd1c098e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:02.307397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79576e6e-86ac-4fa6-8024-4724cd158a2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:02.308201 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79576e6e-86ac-4fa6-8024-4724cd158a2c admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:02.308541 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79576e6e-86ac-4fa6-8024-4724cd158a2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:02.308829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79576e6e-86ac-4fa6-8024-4724cd158a2c admin admin] PATH_INFO: `/v3/projects/996d26fd06a943258239da182d539a21/users/e9bee6efcb1c4053a7cc8894a3d289dd/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:02.310969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79576e6e-86ac-4fa6-8024-4724cd158a2c admin admin] RBAC: Authorizing `identity:create_grant(project_id=996d26fd06a943258239da182d539a21, user_id=e9bee6efcb1c4053a7cc8894a3d289dd, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:02.314668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5939162b-5ca4-48df-a6a2-f7b165bea18b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:02.314956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79576e6e-86ac-4fa6-8024-4724cd158a2c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:02.315338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5939162b-5ca4-48df-a6a2-f7b165bea18b admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:02.315622 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5939162b-5ca4-48df-a6a2-f7b165bea18b admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:02.315896 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5939162b-5ca4-48df-a6a2-f7b165bea18b admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:02.316551 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5939162b-5ca4-48df-a6a2-f7b165bea18b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:02.318453 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5939162b-5ca4-48df-a6a2-f7b165bea18b admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:02.344559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 754/1544] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:02 2022] PUT /identity/v3/projects/996d26fd06a943258239da182d539a21/users/e9bee6efcb1c4053a7cc8894a3d289dd/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:02.344559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6086ce8-ab59-4879-85fd-0a476ecb0960 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:02.344559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6086ce8-ab59-4879-85fd-0a476ecb0960 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:02.344559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6086ce8-ab59-4879-85fd-0a476ecb0960 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:02.349281 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 791/1545] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:02 2022] GET /identity/v3/roles => generated 1420 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:02.350723 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5939162b-5ca4-48df-a6a2-f7b165bea18b admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:02.374493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5939162b-5ca4-48df-a6a2-f7b165bea18b admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:02.401448 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d6086ce8-ab59-4879-85fd-0a476ecb0960 None None] MFA Rules not processed for user `a4dd8f6f10c741fbbce49c6975e13256`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:02.403296 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d6086ce8-ab59-4879-85fd-0a476ecb0960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:02.471143 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a50db7e-c082-4190-870d-dbb39252a7da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:02.471788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a50db7e-c082-4190-870d-dbb39252a7da admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:02.472070 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a50db7e-c082-4190-870d-dbb39252a7da admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:02.472392 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a50db7e-c082-4190-870d-dbb39252a7da admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:02.473051 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a50db7e-c082-4190-870d-dbb39252a7da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:02.474691 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a50db7e-c082-4190-870d-dbb39252a7da admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:02.495417 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 792/1546] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:02 2022] GET /identity/v3/roles => generated 1420 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:02.495417 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392654af-cfab-4b1e-912f-1c953f6af9f3 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:02.495417 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392654af-cfab-4b1e-912f-1c953f6af9f3 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:02.495417 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392654af-cfab-4b1e-912f-1c953f6af9f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:02.526214 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 755/1547] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:02 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:02.526214 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6086ce8-ab59-4879-85fd-0a476ecb0960 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:02.532831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-392654af-cfab-4b1e-912f-1c953f6af9f3 None None] MFA Rules not processed for user `e9bee6efcb1c4053a7cc8894a3d289dd`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:02.534171 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-392654af-cfab-4b1e-912f-1c953f6af9f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:02.535141 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6086ce8-ab59-4879-85fd-0a476ecb0960 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:02.539668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:02.540077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d32b262-3cc5-4c60-b6f5-c6df55fa25e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:02.541178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d32b262-3cc5-4c60-b6f5-c6df55fa25e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:02.541178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d32b262-3cc5-4c60-b6f5-c6df55fa25e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:02.541434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d32b262-3cc5-4c60-b6f5-c6df55fa25e1 admin admin] PATH_INFO: `/v3/projects/f88dc43c613e45aab8456768d988fb0c/users/584927648cab49af9d38ad8d304e5ee1/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:02.544688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d32b262-3cc5-4c60-b6f5-c6df55fa25e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f88dc43c613e45aab8456768d988fb0c, user_id=584927648cab49af9d38ad8d304e5ee1, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:02.547624 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d32b262-3cc5-4c60-b6f5-c6df55fa25e1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:02.569565 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 756/1548] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:02 2022] PUT /identity/v3/projects/f88dc43c613e45aab8456768d988fb0c/users/584927648cab49af9d38ad8d304e5ee1/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:02.570507 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d32b262-3cc5-4c60-b6f5-c6df55fa25e1 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:02.620621 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 793/1549] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:02 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:02.622656 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392654af-cfab-4b1e-912f-1c953f6af9f3 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:02.624708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-392654af-cfab-4b1e-912f-1c953f6af9f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:02.628759 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:02.677071 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d32b262-3cc5-4c60-b6f5-c6df55fa25e1 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:02.684060 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88fb616-618f-4068-85ba-5dcb3912c778 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:02.684939 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88fb616-618f-4068-85ba-5dcb3912c778 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:02.685335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88fb616-618f-4068-85ba-5dcb3912c778 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:02.685616 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88fb616-618f-4068-85ba-5dcb3912c778 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:02.686333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88fb616-618f-4068-85ba-5dcb3912c778 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:02.689085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88fb616-618f-4068-85ba-5dcb3912c778 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:02.716348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392654af-cfab-4b1e-912f-1c953f6af9f3 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:02.724595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 757/1550] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:02 2022] GET /identity/v3/roles => generated 1420 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:02.729484 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59efd3b2-43f9-46d7-8513-48279ebfa3b2 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:02.729789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59efd3b2-43f9-46d7-8513-48279ebfa3b2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:02.730034 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59efd3b2-43f9-46d7-8513-48279ebfa3b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:02.761549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-59efd3b2-43f9-46d7-8513-48279ebfa3b2 None None] MFA Rules not processed for user `584927648cab49af9d38ad8d304e5ee1`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:02.763616 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-59efd3b2-43f9-46d7-8513-48279ebfa3b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:02.787947 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4aa2b2-f2ab-42c4-a255-26bd4a883a3c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:02.789048 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4aa2b2-f2ab-42c4-a255-26bd4a883a3c service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:02.789337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4aa2b2-f2ab-42c4-a255-26bd4a883a3c service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:02.789614 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4aa2b2-f2ab-42c4-a255-26bd4a883a3c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:02.874469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 758/1551] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:02 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:02.875921 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59efd3b2-43f9-46d7-8513-48279ebfa3b2 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:02.877954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-59efd3b2-43f9-46d7-8513-48279ebfa3b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:02.885189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4aa2b2-f2ab-42c4-a255-26bd4a883a3c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:02.889083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4aa2b2-f2ab-42c4-a255-26bd4a883a3c service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:02.895710 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59efd3b2-43f9-46d7-8513-48279ebfa3b2 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:02.904117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2e06c8-bb3e-4e39-80ad-dadb6e506c9f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:02.905042 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2e06c8-bb3e-4e39-80ad-dadb6e506c9f service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:02.905400 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2e06c8-bb3e-4e39-80ad-dadb6e506c9f service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:02.905763 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2e06c8-bb3e-4e39-80ad-dadb6e506c9f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:02.907403 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 794/1552] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:02 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:02.908757 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4aa2b2-f2ab-42c4-a255-26bd4a883a3c service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:02.948670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4aa2b2-f2ab-42c4-a255-26bd4a883a3c service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:02.955141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72484af5-b17a-4a2d-ad6f-09ae8ab9f74b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:02.956223 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72484af5-b17a-4a2d-ad6f-09ae8ab9f74b admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:02.956580 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72484af5-b17a-4a2d-ad6f-09ae8ab9f74b admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:02.956928 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72484af5-b17a-4a2d-ad6f-09ae8ab9f74b admin admin] PATH_INFO: `/v3/projects/69722a0b3c1440de98c6e392d1b120c7/users/d7ec1d1c4dae476eaa110e2b6dc509e3/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:02.961048 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:02.979506 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72484af5-b17a-4a2d-ad6f-09ae8ab9f74b admin admin] RBAC: Authorizing `identity:create_grant(project_id=69722a0b3c1440de98c6e392d1b120c7, user_id=d7ec1d1c4dae476eaa110e2b6dc509e3, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:02.983007 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72484af5-b17a-4a2d-ad6f-09ae8ab9f74b admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:03.004902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 795/1553] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:02 2022] PUT /identity/v3/projects/69722a0b3c1440de98c6e392d1b120c7/users/d7ec1d1c4dae476eaa110e2b6dc509e3/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:03.006594 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72484af5-b17a-4a2d-ad6f-09ae8ab9f74b admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:03.053270 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2e06c8-bb3e-4e39-80ad-dadb6e506c9f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:03.054749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2e06c8-bb3e-4e39-80ad-dadb6e506c9f service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:03.070238 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 759/1554] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:02 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:03.072433 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2b8ba6-2401-4120-b687-4543979143c2 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:03.072946 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2b8ba6-2401-4120-b687-4543979143c2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:03.073427 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2b8ba6-2401-4120-b687-4543979143c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:03.127675 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2b2b8ba6-2401-4120-b687-4543979143c2 None None] MFA Rules not processed for user `d7ec1d1c4dae476eaa110e2b6dc509e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:03.129483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2b2b8ba6-2401-4120-b687-4543979143c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:03.155910 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72484af5-b17a-4a2d-ad6f-09ae8ab9f74b admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:03.166179 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6066c2db-2c0b-413b-b36e-78fe0360d49c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:03.168315 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6066c2db-2c0b-413b-b36e-78fe0360d49c service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:03.168734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6066c2db-2c0b-413b-b36e-78fe0360d49c service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:03.169019 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6066c2db-2c0b-413b-b36e-78fe0360d49c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:03.188731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:03.296996 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 760/1555] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:03 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:03.302180 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2b8ba6-2401-4120-b687-4543979143c2 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:03.309352 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6066c2db-2c0b-413b-b36e-78fe0360d49c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:03.312281 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6066c2db-2c0b-413b-b36e-78fe0360d49c service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:03.337287 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 796/1556] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:03 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:03.338726 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6066c2db-2c0b-413b-b36e-78fe0360d49c service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:03.385468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2b8ba6-2401-4120-b687-4543979143c2 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:03.391795 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f86b24-2c30-4578-b1a3-c631a000157d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:03.393292 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f86b24-2c30-4578-b1a3-c631a000157d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:03.393640 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f86b24-2c30-4578-b1a3-c631a000157d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:03.394042 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f86b24-2c30-4578-b1a3-c631a000157d admin admin] PATH_INFO: `/v3/users/d1df414c5cf94f2c8ee44ad1b01b6ac6` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:03.395585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f86b24-2c30-4578-b1a3-c631a000157d admin admin] RBAC: Authorizing `identity:delete_user(user_id=d1df414c5cf94f2c8ee44ad1b01b6ac6)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:03.397623 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f86b24-2c30-4578-b1a3-c631a000157d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:03.486317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6066c2db-2c0b-413b-b36e-78fe0360d49c service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:03.506379 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bfe372-3ff2-4090-89e4-0ca10a5fab24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:03.507122 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bfe372-3ff2-4090-89e4-0ca10a5fab24 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:03.507422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bfe372-3ff2-4090-89e4-0ca10a5fab24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:03.507683 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bfe372-3ff2-4090-89e4-0ca10a5fab24 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:03.508352 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bfe372-3ff2-4090-89e4-0ca10a5fab24 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:03.510702 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bfe372-3ff2-4090-89e4-0ca10a5fab24 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:03.518485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e1f86b24-2c30-4578-b1a3-c631a000157d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd1df414c5cf94f2c8ee44ad1b01b6ac6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:03.519285 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e1f86b24-2c30-4578-b1a3-c631a000157d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd1df414c5cf94f2c8ee44ad1b01b6ac6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:03.530300 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e1f86b24-2c30-4578-b1a3-c631a000157d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd1df414c5cf94f2c8ee44ad1b01b6ac6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:03.531151 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e1f86b24-2c30-4578-b1a3-c631a000157d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd1df414c5cf94f2c8ee44ad1b01b6ac6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:03.552163 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e1f86b24-2c30-4578-b1a3-c631a000157d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd1df414c5cf94f2c8ee44ad1b01b6ac6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:03.575279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 797/1557] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:03 2022] POST /identity/v3/projects => generated 387 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:03.580376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bfe372-3ff2-4090-89e4-0ca10a5fab24 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:03.595397 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-63bfe372-3ff2-4090-89e4-0ca10a5fab24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:03.599874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 761/1558] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:03 2022] DELETE /identity/v3/users/d1df414c5cf94f2c8ee44ad1b01b6ac6 => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:03.601359 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f86b24-2c30-4578-b1a3-c631a000157d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:03.603142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e1f86b24-2c30-4578-b1a3-c631a000157d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:03.631005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:03.702751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bfe372-3ff2-4090-89e4-0ca10a5fab24 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:03.705228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:03.712927 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81257b01-c574-4a3b-9b42-b58638566b3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:03.714720 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81257b01-c574-4a3b-9b42-b58638566b3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:03.715644 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81257b01-c574-4a3b-9b42-b58638566b3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:03.716236 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81257b01-c574-4a3b-9b42-b58638566b3a admin admin] PATH_INFO: `/v3/users/b484e5d8140d4bc2894c663466f47c61` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:03.718081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81257b01-c574-4a3b-9b42-b58638566b3a admin admin] RBAC: Authorizing `identity:delete_user(user_id=b484e5d8140d4bc2894c663466f47c61)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:03.718910 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f86b24-2c30-4578-b1a3-c631a000157d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:03.721907 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81257b01-c574-4a3b-9b42-b58638566b3a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:03.724515 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd032f6-7cea-4b01-a07c-c293ec0a4f96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:03.725174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd032f6-7cea-4b01-a07c-c293ec0a4f96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:03.725541 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd032f6-7cea-4b01-a07c-c293ec0a4f96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:03.726250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd032f6-7cea-4b01-a07c-c293ec0a4f96 admin admin] PATH_INFO: `/v3/users/71b22c2c96994a5a89749a7cb7753b66` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:03.727306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd032f6-7cea-4b01-a07c-c293ec0a4f96 admin admin] RBAC: Authorizing `identity:delete_user(user_id=71b22c2c96994a5a89749a7cb7753b66)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:03.732303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd032f6-7cea-4b01-a07c-c293ec0a4f96 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:03.792394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4cd032f6-7cea-4b01-a07c-c293ec0a4f96 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '71b22c2c96994a5a89749a7cb7753b66'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:03.793584 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4cd032f6-7cea-4b01-a07c-c293ec0a4f96 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '71b22c2c96994a5a89749a7cb7753b66'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:03.805286 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-81257b01-c574-4a3b-9b42-b58638566b3a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b484e5d8140d4bc2894c663466f47c61'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:03.806044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-81257b01-c574-4a3b-9b42-b58638566b3a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b484e5d8140d4bc2894c663466f47c61'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:03.807414 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4cd032f6-7cea-4b01-a07c-c293ec0a4f96 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '71b22c2c96994a5a89749a7cb7753b66'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:03.808289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4cd032f6-7cea-4b01-a07c-c293ec0a4f96 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '71b22c2c96994a5a89749a7cb7753b66'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:03.816991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-81257b01-c574-4a3b-9b42-b58638566b3a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b484e5d8140d4bc2894c663466f47c61'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:03.827779 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4cd032f6-7cea-4b01-a07c-c293ec0a4f96 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '71b22c2c96994a5a89749a7cb7753b66'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:03.839270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-81257b01-c574-4a3b-9b42-b58638566b3a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b484e5d8140d4bc2894c663466f47c61'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:03.850987 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 762/1559] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:03 2022] DELETE /identity/v3/users/71b22c2c96994a5a89749a7cb7753b66 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:03.851219 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-81257b01-c574-4a3b-9b42-b58638566b3a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b484e5d8140d4bc2894c663466f47c61'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:03.852267 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd032f6-7cea-4b01-a07c-c293ec0a4f96 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:03.854114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4cd032f6-7cea-4b01-a07c-c293ec0a4f96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:03.864029 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 798/1560] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:03 2022] DELETE /identity/v3/users/b484e5d8140d4bc2894c663466f47c61 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:03.866230 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d60240c-46e2-43d9-877f-2103535ea737 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:03.866528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d60240c-46e2-43d9-877f-2103535ea737 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:03.866813 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d60240c-46e2-43d9-877f-2103535ea737 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:03.907155 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-0d60240c-46e2-43d9-877f-2103535ea737 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:03.910302 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-0d60240c-46e2-43d9-877f-2103535ea737 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:03.945961 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd032f6-7cea-4b01-a07c-c293ec0a4f96 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:04.004097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 799/1561] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:04.007383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465cae42-2afc-4df9-b4dd-4c960e032936 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:04.007755 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465cae42-2afc-4df9-b4dd-4c960e032936 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:04.008795 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465cae42-2afc-4df9-b4dd-4c960e032936 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:04.011277 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 800/1562] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 16 00:58:04 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 00:58:04.013209 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddc1e17-eaf9-4cf0-8703-3a2c00daf956 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:04.013503 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddc1e17-eaf9-4cf0-8703-3a2c00daf956 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:04.013775 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddc1e17-eaf9-4cf0-8703-3a2c00daf956 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:04.041836 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-cddc1e17-eaf9-4cf0-8703-3a2c00daf956 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:04.043628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-cddc1e17-eaf9-4cf0-8703-3a2c00daf956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:04.044541 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fae5df-ee81-447b-972c-a87c26d7c1fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:04.046705 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fae5df-ee81-447b-972c-a87c26d7c1fa admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:04.049782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fae5df-ee81-447b-972c-a87c26d7c1fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:04.050161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fae5df-ee81-447b-972c-a87c26d7c1fa admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:04.052680 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0fae5df-ee81-447b-972c-a87c26d7c1fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:04.055011 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 801/1563] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:04.057210 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf2f5d6-755f-4aa4-a3a7-6f63ef063b3b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:04.057762 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf2f5d6-755f-4aa4-a3a7-6f63ef063b3b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:04.058415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf2f5d6-755f-4aa4-a3a7-6f63ef063b3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:04.061481 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0fae5df-ee81-447b-972c-a87c26d7c1fa admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:04.114655 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 763/1564] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:03 2022] POST /identity/v3/users => generated 294 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:04.116653 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fae5df-ee81-447b-972c-a87c26d7c1fa admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:04.118683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c0fae5df-ee81-447b-972c-a87c26d7c1fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:04.130781 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-0bf2f5d6-755f-4aa4-a3a7-6f63ef063b3b None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:04.133134 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-0bf2f5d6-755f-4aa4-a3a7-6f63ef063b3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:04.160993 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:04.163483 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 802/1565] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:04.165254 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d611dc-db4f-4e27-a7b7-47dbf6b40b72 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:04.165877 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d611dc-db4f-4e27-a7b7-47dbf6b40b72 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:04.166296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d611dc-db4f-4e27-a7b7-47dbf6b40b72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:04.196923 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-25d611dc-db4f-4e27-a7b7-47dbf6b40b72 None None] MFA Rules not processed for user `4eba26813e154595a12a67fe4be5eeff`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:04.198991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-25d611dc-db4f-4e27-a7b7-47dbf6b40b72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:04.234997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:04.240427 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fae5df-ee81-447b-972c-a87c26d7c1fa admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:04.245410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf55ddb-a598-4882-bc5e-1cd5fa2098ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:04.246169 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf55ddb-a598-4882-bc5e-1cd5fa2098ac service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:04.246406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf55ddb-a598-4882-bc5e-1cd5fa2098ac service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:04.246679 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf55ddb-a598-4882-bc5e-1cd5fa2098ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:04.264512 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf55ddb-a598-4882-bc5e-1cd5fa2098ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:04.266165 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf55ddb-a598-4882-bc5e-1cd5fa2098ac service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:04.297256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 764/1566] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:04.299629 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf55ddb-a598-4882-bc5e-1cd5fa2098ac service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:04.313854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf55ddb-a598-4882-bc5e-1cd5fa2098ac service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:04.318039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00092d3c-638c-4235-9acd-0e18447de687 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:04.318667 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00092d3c-638c-4235-9acd-0e18447de687 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:04.318981 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00092d3c-638c-4235-9acd-0e18447de687 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:04.319258 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00092d3c-638c-4235-9acd-0e18447de687 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:04.325459 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 803/1567] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 16 00:58:04 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:04.326779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d611dc-db4f-4e27-a7b7-47dbf6b40b72 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:04.328521 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-25d611dc-db4f-4e27-a7b7-47dbf6b40b72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:04.333145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00092d3c-638c-4235-9acd-0e18447de687 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:04.335174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00092d3c-638c-4235-9acd-0e18447de687 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:04.347285 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d611dc-db4f-4e27-a7b7-47dbf6b40b72 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:04.351389 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 765/1568] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:04.352665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00092d3c-638c-4235-9acd-0e18447de687 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:04.354879 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d13856f-dafa-4291-9938-de21f8656796 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:04.356016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d13856f-dafa-4291-9938-de21f8656796 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:04.356417 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-00092d3c-638c-4235-9acd-0e18447de687 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:04.356567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d13856f-dafa-4291-9938-de21f8656796 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:04.356879 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d13856f-dafa-4291-9938-de21f8656796 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:04.357690 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d13856f-dafa-4291-9938-de21f8656796 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:04.359966 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d13856f-dafa-4291-9938-de21f8656796 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:04.372103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00092d3c-638c-4235-9acd-0e18447de687 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:04.389751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 804/1569] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:04 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:04.389751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d13856f-dafa-4291-9938-de21f8656796 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:04.390770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4d13856f-dafa-4291-9938-de21f8656796 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:04.409378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8088fbcc-e263-446c-82a1-0e9503c5a74f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:04.410314 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8088fbcc-e263-446c-82a1-0e9503c5a74f service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:04.411774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8088fbcc-e263-446c-82a1-0e9503c5a74f service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:04.412838 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8088fbcc-e263-446c-82a1-0e9503c5a74f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:04.429475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d13856f-dafa-4291-9938-de21f8656796 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:04.436549 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647b78e4-ec92-4a62-9aea-c2a6ecd3f3de service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:04.437724 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647b78e4-ec92-4a62-9aea-c2a6ecd3f3de service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:04.438262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647b78e4-ec92-4a62-9aea-c2a6ecd3f3de service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:04.438815 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647b78e4-ec92-4a62-9aea-c2a6ecd3f3de service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:04.454779 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8088fbcc-e263-446c-82a1-0e9503c5a74f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:04.456664 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647b78e4-ec92-4a62-9aea-c2a6ecd3f3de service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:04.457952 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8088fbcc-e263-446c-82a1-0e9503c5a74f service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:04.458673 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647b78e4-ec92-4a62-9aea-c2a6ecd3f3de service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:04.478560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 805/1570] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:04.480010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647b78e4-ec92-4a62-9aea-c2a6ecd3f3de service nova] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:04.490094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 766/1571] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:04.496437 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647b78e4-ec92-4a62-9aea-c2a6ecd3f3de service nova] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:04.503553 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e12cc0-d3d6-4800-a5c6-a454d1d18258 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:04.504485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e12cc0-d3d6-4800-a5c6-a454d1d18258 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:04.504926 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e12cc0-d3d6-4800-a5c6-a454d1d18258 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:04.505601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e12cc0-d3d6-4800-a5c6-a454d1d18258 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:04.506420 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e12cc0-d3d6-4800-a5c6-a454d1d18258 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:04.508519 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e12cc0-d3d6-4800-a5c6-a454d1d18258 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:04.526649 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 806/1572] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:04 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:04.527506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8088fbcc-e263-446c-82a1-0e9503c5a74f service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:04.545653 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8088fbcc-e263-446c-82a1-0e9503c5a74f service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:04.555892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f75276-e89d-4202-990b-34e8974dfabd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:04.557060 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f75276-e89d-4202-990b-34e8974dfabd admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:04.557879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f75276-e89d-4202-990b-34e8974dfabd admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:04.558162 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f75276-e89d-4202-990b-34e8974dfabd admin admin] PATH_INFO: `/v3/projects/039a813f1de5494cbd44cc85ede9fbf8/users/9ee9c70f18b74717b51f4230940e3aad/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:04.581773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f75276-e89d-4202-990b-34e8974dfabd admin admin] RBAC: Authorizing `identity:create_grant(project_id=039a813f1de5494cbd44cc85ede9fbf8, user_id=9ee9c70f18b74717b51f4230940e3aad, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:04.585299 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f75276-e89d-4202-990b-34e8974dfabd admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:04.620198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 767/1573] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:04 2022] PUT /identity/v3/projects/039a813f1de5494cbd44cc85ede9fbf8/users/9ee9c70f18b74717b51f4230940e3aad/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:58:04.634406 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6084ae1a-6914-44a5-9649-469ad9ed7d40 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:04.634811 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6084ae1a-6914-44a5-9649-469ad9ed7d40 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:04.635320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6084ae1a-6914-44a5-9649-469ad9ed7d40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:04.702428 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-6084ae1a-6914-44a5-9649-469ad9ed7d40 None None] MFA Rules not processed for user `9ee9c70f18b74717b51f4230940e3aad`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:04.704027 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6084ae1a-6914-44a5-9649-469ad9ed7d40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:04.857029 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 807/1574] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:04 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:04.858317 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f75276-e89d-4202-990b-34e8974dfabd admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:04.969676 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f75276-e89d-4202-990b-34e8974dfabd admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:04.975780 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052c5291-76fd-4db8-aa05-57b85ca8586d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:04.976625 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052c5291-76fd-4db8-aa05-57b85ca8586d admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:04.976953 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052c5291-76fd-4db8-aa05-57b85ca8586d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:04.977386 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052c5291-76fd-4db8-aa05-57b85ca8586d admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:04.978457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052c5291-76fd-4db8-aa05-57b85ca8586d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:04.982121 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052c5291-76fd-4db8-aa05-57b85ca8586d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:05.031816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 768/1575] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:04 2022] POST /identity/v3/projects => generated 387 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:58:05.040444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6084ae1a-6914-44a5-9649-469ad9ed7d40 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:05.065579 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6084ae1a-6914-44a5-9649-469ad9ed7d40 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:05.141482 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052c5291-76fd-4db8-aa05-57b85ca8586d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:05.143459 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-052c5291-76fd-4db8-aa05-57b85ca8586d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:05.197526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f61b40-f1fe-47d8-94e7-588bd2e81433 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:05.198200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f61b40-f1fe-47d8-94e7-588bd2e81433 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:05.198484 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f61b40-f1fe-47d8-94e7-588bd2e81433 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:05.198759 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f61b40-f1fe-47d8-94e7-588bd2e81433 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:05.200133 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f61b40-f1fe-47d8-94e7-588bd2e81433 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:05.204096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f61b40-f1fe-47d8-94e7-588bd2e81433 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:05.259833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 808/1576] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:05 2022] POST /identity/v3/users => generated 300 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:05.266488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f61b40-f1fe-47d8-94e7-588bd2e81433 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:05.272393 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:05.274406 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:05.285326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f61b40-f1fe-47d8-94e7-588bd2e81433 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:05.292019 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dad74e8-ad95-49f9-928f-5a19208b36cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:05.292768 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dad74e8-ad95-49f9-928f-5a19208b36cc admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:05.293306 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dad74e8-ad95-49f9-928f-5a19208b36cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:05.293702 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dad74e8-ad95-49f9-928f-5a19208b36cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:05.294776 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dad74e8-ad95-49f9-928f-5a19208b36cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:05.297133 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dad74e8-ad95-49f9-928f-5a19208b36cc admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:05.319377 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052c5291-76fd-4db8-aa05-57b85ca8586d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:05.323082 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 809/1577] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:05 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:05.324597 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dad74e8-ad95-49f9-928f-5a19208b36cc admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:05.340751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a14a1f-0577-45a7-bfa3-0db2042c275c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:05.341963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a14a1f-0577-45a7-bfa3-0db2042c275c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:05.342910 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a14a1f-0577-45a7-bfa3-0db2042c275c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:05.342910 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a14a1f-0577-45a7-bfa3-0db2042c275c admin admin] PATH_INFO: `/v3/projects/35ea763c74bf4281a2492556062f5b78` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:05.344120 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a14a1f-0577-45a7-bfa3-0db2042c275c admin admin] RBAC: Authorizing `identity:delete_project(project_id=35ea763c74bf4281a2492556062f5b78)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:05.346496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a14a1f-0577-45a7-bfa3-0db2042c275c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:05.359692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dad74e8-ad95-49f9-928f-5a19208b36cc admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:05.370831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b23b37d-d0f7-43e6-8d0e-0c08b89b7d0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:05.375699 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b23b37d-d0f7-43e6-8d0e-0c08b89b7d0b admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:05.377467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b23b37d-d0f7-43e6-8d0e-0c08b89b7d0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:05.378091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b23b37d-d0f7-43e6-8d0e-0c08b89b7d0b admin admin] PATH_INFO: `/v3/projects/978a7bd015b148ff81040c266cdcf551/users/d076aa56efb748f0becaea215cf915b7/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:05.422968 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b23b37d-d0f7-43e6-8d0e-0c08b89b7d0b admin admin] RBAC: Authorizing `identity:create_grant(project_id=978a7bd015b148ff81040c266cdcf551, user_id=d076aa56efb748f0becaea215cf915b7, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:05.430927 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b23b37d-d0f7-43e6-8d0e-0c08b89b7d0b admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:05.441328 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-28a14a1f-0577-45a7-bfa3-0db2042c275c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '35ea763c74bf4281a2492556062f5b78', 'request_id': 'req-28a14a1f-0577-45a7-bfa3-0db2042c275c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:05.453771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 810/1578] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:05 2022] PUT /identity/v3/projects/978a7bd015b148ff81040c266cdcf551/users/d076aa56efb748f0becaea215cf915b7/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:05.459505 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-28a14a1f-0577-45a7-bfa3-0db2042c275c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '35ea763c74bf4281a2492556062f5b78', 'request_id': 'req-28a14a1f-0577-45a7-bfa3-0db2042c275c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:05.472161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b23b37d-d0f7-43e6-8d0e-0c08b89b7d0b admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:05.482592 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-28a14a1f-0577-45a7-bfa3-0db2042c275c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '35ea763c74bf4281a2492556062f5b78', 'request_id': 'req-28a14a1f-0577-45a7-bfa3-0db2042c275c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:05.511095 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-28a14a1f-0577-45a7-bfa3-0db2042c275c admin admin] The token cache is being invalidate because project 35ea763c74bf4281a2492556062f5b78 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:05.511708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-28a14a1f-0577-45a7-bfa3-0db2042c275c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:05.512831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-28a14a1f-0577-45a7-bfa3-0db2042c275c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:05.520525 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 769/1579] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:05 2022] DELETE /identity/v3/projects/35ea763c74bf4281a2492556062f5b78 => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:58:05.534209 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a14a1f-0577-45a7-bfa3-0db2042c275c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:05.539391 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-28a14a1f-0577-45a7-bfa3-0db2042c275c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:05.610185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b23b37d-d0f7-43e6-8d0e-0c08b89b7d0b admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:05.617366 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c92470a-e22e-4b47-b697-5bfb18060f69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:05.618439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c92470a-e22e-4b47-b697-5bfb18060f69 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:05.619048 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c92470a-e22e-4b47-b697-5bfb18060f69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:05.619567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c92470a-e22e-4b47-b697-5bfb18060f69 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:05.630200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c92470a-e22e-4b47-b697-5bfb18060f69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:05.633029 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c92470a-e22e-4b47-b697-5bfb18060f69 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:05.652870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 811/1580] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:05 2022] GET /identity/v3/roles => generated 1420 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:05.655043 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c92470a-e22e-4b47-b697-5bfb18060f69 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:05.658646 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-7c92470a-e22e-4b47-b697-5bfb18060f69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:05.666471 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a14a1f-0577-45a7-bfa3-0db2042c275c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:05.679326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c92470a-e22e-4b47-b697-5bfb18060f69 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:05.684080 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0f7ebc-3792-49cd-97ee-9c009b55c020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:05.685290 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0f7ebc-3792-49cd-97ee-9c009b55c020 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:05.686121 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f852735-acc2-4c1f-802c-d3ebb137db35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:05.687294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f852735-acc2-4c1f-802c-d3ebb137db35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:05.688232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f852735-acc2-4c1f-802c-d3ebb137db35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:05.688548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0f7ebc-3792-49cd-97ee-9c009b55c020 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:05.688666 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f852735-acc2-4c1f-802c-d3ebb137db35 admin admin] PATH_INFO: `/v3/projects/8d819a5b4ad146bb8c3eccf8c8e2d7e9` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:05.689591 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0f7ebc-3792-49cd-97ee-9c009b55c020 admin admin] PATH_INFO: `/v3/users/e9bee6efcb1c4053a7cc8894a3d289dd` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:05.690236 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f852735-acc2-4c1f-802c-d3ebb137db35 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d819a5b4ad146bb8c3eccf8c8e2d7e9)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:05.692791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f852735-acc2-4c1f-802c-d3ebb137db35 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:05.695014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0f7ebc-3792-49cd-97ee-9c009b55c020 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e9bee6efcb1c4053a7cc8894a3d289dd)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:05.703895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0f7ebc-3792-49cd-97ee-9c009b55c020 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:05.801868 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-7f852735-acc2-4c1f-802c-d3ebb137db35 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d819a5b4ad146bb8c3eccf8c8e2d7e9', 'request_id': 'req-7f852735-acc2-4c1f-802c-d3ebb137db35'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:05.815079 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-7f852735-acc2-4c1f-802c-d3ebb137db35 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d819a5b4ad146bb8c3eccf8c8e2d7e9', 'request_id': 'req-7f852735-acc2-4c1f-802c-d3ebb137db35'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:05.835407 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-7f852735-acc2-4c1f-802c-d3ebb137db35 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d819a5b4ad146bb8c3eccf8c8e2d7e9', 'request_id': 'req-7f852735-acc2-4c1f-802c-d3ebb137db35'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:05.874919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-da0f7ebc-3792-49cd-97ee-9c009b55c020 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e9bee6efcb1c4053a7cc8894a3d289dd'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:05.876792 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-da0f7ebc-3792-49cd-97ee-9c009b55c020 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e9bee6efcb1c4053a7cc8894a3d289dd'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:05.898446 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-da0f7ebc-3792-49cd-97ee-9c009b55c020 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e9bee6efcb1c4053a7cc8894a3d289dd'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:05.900585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-da0f7ebc-3792-49cd-97ee-9c009b55c020 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e9bee6efcb1c4053a7cc8894a3d289dd'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:05.941196 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-da0f7ebc-3792-49cd-97ee-9c009b55c020 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e9bee6efcb1c4053a7cc8894a3d289dd'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:05.963172 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:05.977988 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 770/1581] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:05 2022] DELETE /identity/v3/users/e9bee6efcb1c4053a7cc8894a3d289dd => generated 0 bytes in 446 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:05.979580 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0f7ebc-3792-49cd-97ee-9c009b55c020 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:05.985011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-da0f7ebc-3792-49cd-97ee-9c009b55c020 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:06.011673 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-7f852735-acc2-4c1f-802c-d3ebb137db35 admin admin] The token cache is being invalidate because project 8d819a5b4ad146bb8c3eccf8c8e2d7e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:06.012343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-7f852735-acc2-4c1f-802c-d3ebb137db35 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.015294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-7f852735-acc2-4c1f-802c-d3ebb137db35 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.020064 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 812/1582] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:05 2022] DELETE /identity/v3/projects/8d819a5b4ad146bb8c3eccf8c8e2d7e9 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:06.021045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f852735-acc2-4c1f-802c-d3ebb137db35 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:06.025897 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-7f852735-acc2-4c1f-802c-d3ebb137db35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:06.103631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0f7ebc-3792-49cd-97ee-9c009b55c020 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:06.109857 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da691a90-42ca-4eab-9bb7-ad56d05addcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:06.110664 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da691a90-42ca-4eab-9bb7-ad56d05addcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:06.111352 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da691a90-42ca-4eab-9bb7-ad56d05addcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:06.111638 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da691a90-42ca-4eab-9bb7-ad56d05addcb admin admin] PATH_INFO: `/v3/projects/56dafe1503724a8387b09e7ced1f22b6` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:06.113709 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da691a90-42ca-4eab-9bb7-ad56d05addcb admin admin] RBAC: Authorizing `identity:delete_project(project_id=56dafe1503724a8387b09e7ced1f22b6)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:06.120145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da691a90-42ca-4eab-9bb7-ad56d05addcb admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:06.203271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-da691a90-42ca-4eab-9bb7-ad56d05addcb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '56dafe1503724a8387b09e7ced1f22b6', 'request_id': 'req-da691a90-42ca-4eab-9bb7-ad56d05addcb'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.209691 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:06.222908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-da691a90-42ca-4eab-9bb7-ad56d05addcb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '56dafe1503724a8387b09e7ced1f22b6', 'request_id': 'req-da691a90-42ca-4eab-9bb7-ad56d05addcb'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.247459 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-da691a90-42ca-4eab-9bb7-ad56d05addcb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '56dafe1503724a8387b09e7ced1f22b6', 'request_id': 'req-da691a90-42ca-4eab-9bb7-ad56d05addcb'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.264370 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f852735-acc2-4c1f-802c-d3ebb137db35 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:06.276589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-da691a90-42ca-4eab-9bb7-ad56d05addcb admin admin] The token cache is being invalidate because project 56dafe1503724a8387b09e7ced1f22b6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:06.276589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-da691a90-42ca-4eab-9bb7-ad56d05addcb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.276589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-da691a90-42ca-4eab-9bb7-ad56d05addcb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.277039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 771/1583] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:05 2022] DELETE /identity/v3/projects/56dafe1503724a8387b09e7ced1f22b6 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:06.277986 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da691a90-42ca-4eab-9bb7-ad56d05addcb admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:06.282169 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-da691a90-42ca-4eab-9bb7-ad56d05addcb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:06.375614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da691a90-42ca-4eab-9bb7-ad56d05addcb admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:06.381076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4474dbe8-42d5-4fad-96b6-8bef7a60772d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:06.382138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4474dbe8-42d5-4fad-96b6-8bef7a60772d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:06.382521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4474dbe8-42d5-4fad-96b6-8bef7a60772d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:06.382924 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4474dbe8-42d5-4fad-96b6-8bef7a60772d admin admin] PATH_INFO: `/v3/users/584927648cab49af9d38ad8d304e5ee1` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:06.406571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4474dbe8-42d5-4fad-96b6-8bef7a60772d admin admin] RBAC: Authorizing `identity:delete_user(user_id=584927648cab49af9d38ad8d304e5ee1)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:06.406571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4474dbe8-42d5-4fad-96b6-8bef7a60772d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:06.438291 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e6d58e-e52c-4a85-bfaa-d133f002919c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:06.440109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e6d58e-e52c-4a85-bfaa-d133f002919c admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:06.440604 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e6d58e-e52c-4a85-bfaa-d133f002919c admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:06.441032 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e6d58e-e52c-4a85-bfaa-d133f002919c admin admin] PATH_INFO: `/v3/projects/978a7bd015b148ff81040c266cdcf551/users/d076aa56efb748f0becaea215cf915b7/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:06.443486 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e6d58e-e52c-4a85-bfaa-d133f002919c admin admin] RBAC: Authorizing `identity:create_grant(project_id=978a7bd015b148ff81040c266cdcf551, user_id=d076aa56efb748f0becaea215cf915b7, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:06.446408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e6d58e-e52c-4a85-bfaa-d133f002919c admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:06.461804 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 813/1584] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:06 2022] PUT /identity/v3/projects/978a7bd015b148ff81040c266cdcf551/users/d076aa56efb748f0becaea215cf915b7/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:06.463153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e6d58e-e52c-4a85-bfaa-d133f002919c admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:06.465150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-37e6d58e-e52c-4a85-bfaa-d133f002919c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:06.526757 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:06.567631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4474dbe8-42d5-4fad-96b6-8bef7a60772d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '584927648cab49af9d38ad8d304e5ee1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.568421 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4474dbe8-42d5-4fad-96b6-8bef7a60772d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '584927648cab49af9d38ad8d304e5ee1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.582139 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4474dbe8-42d5-4fad-96b6-8bef7a60772d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '584927648cab49af9d38ad8d304e5ee1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.582847 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4474dbe8-42d5-4fad-96b6-8bef7a60772d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '584927648cab49af9d38ad8d304e5ee1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.601158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4474dbe8-42d5-4fad-96b6-8bef7a60772d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '584927648cab49af9d38ad8d304e5ee1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.620475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e6d58e-e52c-4a85-bfaa-d133f002919c admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:06.625896 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 772/1585] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:06 2022] DELETE /identity/v3/users/584927648cab49af9d38ad8d304e5ee1 => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:06.625896 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338c2375-cd48-4f69-b7aa-29014e56cd94 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:06.625896 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338c2375-cd48-4f69-b7aa-29014e56cd94 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:06.625896 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338c2375-cd48-4f69-b7aa-29014e56cd94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:06.680763 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-338c2375-cd48-4f69-b7aa-29014e56cd94 None None] MFA Rules not processed for user `d076aa56efb748f0becaea215cf915b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:06.682689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-338c2375-cd48-4f69-b7aa-29014e56cd94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:06.719474 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe32093-bd93-4b1c-935c-bd30d8821af8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:06.720938 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe32093-bd93-4b1c-935c-bd30d8821af8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:06.721536 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe32093-bd93-4b1c-935c-bd30d8821af8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:06.722597 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe32093-bd93-4b1c-935c-bd30d8821af8 admin admin] PATH_INFO: `/v3/users/a4dd8f6f10c741fbbce49c6975e13256` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:06.724309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe32093-bd93-4b1c-935c-bd30d8821af8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a4dd8f6f10c741fbbce49c6975e13256)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:06.727804 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe32093-bd93-4b1c-935c-bd30d8821af8 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:06.787384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=94,keystone:INSERT=1,keystone:DELETE=5 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:06.798979 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 773/1586] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:06 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:06.798979 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338c2375-cd48-4f69-b7aa-29014e56cd94 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:06.800000 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-338c2375-cd48-4f69-b7aa-29014e56cd94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:06.810555 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-bfe32093-bd93-4b1c-935c-bd30d8821af8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a4dd8f6f10c741fbbce49c6975e13256'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.811688 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-bfe32093-bd93-4b1c-935c-bd30d8821af8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a4dd8f6f10c741fbbce49c6975e13256'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.825908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338c2375-cd48-4f69-b7aa-29014e56cd94 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:06.828089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-bfe32093-bd93-4b1c-935c-bd30d8821af8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a4dd8f6f10c741fbbce49c6975e13256'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.838176 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3d1244-c660-422d-b903-92a4e501301c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:06.839230 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3d1244-c660-422d-b903-92a4e501301c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:06.840248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3d1244-c660-422d-b903-92a4e501301c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:06.841432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3d1244-c660-422d-b903-92a4e501301c admin admin] PATH_INFO: `/v3/projects/996d26fd06a943258239da182d539a21` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:06.846381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3d1244-c660-422d-b903-92a4e501301c admin admin] RBAC: Authorizing `identity:delete_project(project_id=996d26fd06a943258239da182d539a21)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:06.846672 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-bfe32093-bd93-4b1c-935c-bd30d8821af8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a4dd8f6f10c741fbbce49c6975e13256'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.849820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3d1244-c660-422d-b903-92a4e501301c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:06.855003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-bfe32093-bd93-4b1c-935c-bd30d8821af8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a4dd8f6f10c741fbbce49c6975e13256'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.863421 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 814/1587] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:06 2022] DELETE /identity/v3/users/a4dd8f6f10c741fbbce49c6975e13256 => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:06.864299 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe32093-bd93-4b1c-935c-bd30d8821af8 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:06.865734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-bfe32093-bd93-4b1c-935c-bd30d8821af8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:06.919897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cd3d1244-c660-422d-b903-92a4e501301c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '996d26fd06a943258239da182d539a21', 'request_id': 'req-cd3d1244-c660-422d-b903-92a4e501301c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.930688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cd3d1244-c660-422d-b903-92a4e501301c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '996d26fd06a943258239da182d539a21', 'request_id': 'req-cd3d1244-c660-422d-b903-92a4e501301c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.941773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cd3d1244-c660-422d-b903-92a4e501301c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '996d26fd06a943258239da182d539a21', 'request_id': 'req-cd3d1244-c660-422d-b903-92a4e501301c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.963676 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cd3d1244-c660-422d-b903-92a4e501301c admin admin] The token cache is being invalidate because project 996d26fd06a943258239da182d539a21 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:06.964128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cd3d1244-c660-422d-b903-92a4e501301c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.965569 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cd3d1244-c660-422d-b903-92a4e501301c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:06.968277 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 774/1588] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:06 2022] DELETE /identity/v3/projects/996d26fd06a943258239da182d539a21 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:06.972848 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2053c0b-1983-4945-9e8f-a9a73602a5d1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:06.974719 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2053c0b-1983-4945-9e8f-a9a73602a5d1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:06.976010 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2053c0b-1983-4945-9e8f-a9a73602a5d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:07.015727 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe32093-bd93-4b1c-935c-bd30d8821af8 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:07.031652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c2053c0b-1983-4945-9e8f-a9a73602a5d1 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:07.033618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c2053c0b-1983-4945-9e8f-a9a73602a5d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:07.054093 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:07.115821 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56cf7f1-e0d5-4f46-a092-e0fe4a238fae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:07.118120 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56cf7f1-e0d5-4f46-a092-e0fe4a238fae service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:07.118601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56cf7f1-e0d5-4f46-a092-e0fe4a238fae service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:07.119034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56cf7f1-e0d5-4f46-a092-e0fe4a238fae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:07.123225 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c56cf7f1-e0d5-4f46-a092-e0fe4a238fae service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:07.139656 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 775/1589] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:07.147656 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23534dd2-d77e-41a1-8364-21395ed269be None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:07.148232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23534dd2-d77e-41a1-8364-21395ed269be None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:07.148607 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23534dd2-d77e-41a1-8364-21395ed269be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:07.177434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-23534dd2-d77e-41a1-8364-21395ed269be None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:07.179557 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-23534dd2-d77e-41a1-8364-21395ed269be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:07.189852 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 776/1590] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:07.193695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23534dd2-d77e-41a1-8364-21395ed269be None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:07.203514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23534dd2-d77e-41a1-8364-21395ed269be None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:07.208377 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91314b2-5731-45c0-b482-d35d8339ffe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:07.209000 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91314b2-5731-45c0-b482-d35d8339ffe4 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:07.209283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91314b2-5731-45c0-b482-d35d8339ffe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:07.209561 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91314b2-5731-45c0-b482-d35d8339ffe4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:07.210211 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91314b2-5731-45c0-b482-d35d8339ffe4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:58:07.210519 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91314b2-5731-45c0-b482-d35d8339ffe4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:07.212219 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91314b2-5731-45c0-b482-d35d8339ffe4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:07.227862 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56cf7f1-e0d5-4f46-a092-e0fe4a238fae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:07.230933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56cf7f1-e0d5-4f46-a092-e0fe4a238fae service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:07.231427 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 777/1591] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:58:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:58:07.232655 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91314b2-5731-45c0-b482-d35d8339ffe4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:07.243974 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91314b2-5731-45c0-b482-d35d8339ffe4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:07.246570 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 815/1592] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:06 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:07.249097 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d559ef9-c6f2-4d48-9208-b993a139be38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:07.249831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d559ef9-c6f2-4d48-9208-b993a139be38 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:07.250008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d559ef9-c6f2-4d48-9208-b993a139be38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:07.250303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d559ef9-c6f2-4d48-9208-b993a139be38 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:07.250974 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d559ef9-c6f2-4d48-9208-b993a139be38 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:07.253015 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d559ef9-c6f2-4d48-9208-b993a139be38 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:07.290243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 778/1593] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:07 2022] POST /identity/v3/projects => generated 383 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:07.291870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56cf7f1-e0d5-4f46-a092-e0fe4a238fae service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:07.305161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56cf7f1-e0d5-4f46-a092-e0fe4a238fae service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:07.362517 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:07.468554 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d559ef9-c6f2-4d48-9208-b993a139be38 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:07.471902 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0d559ef9-c6f2-4d48-9208-b993a139be38 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:07.552587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915289f6-6f0f-4f57-9389-9675edda16f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:07.553337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915289f6-6f0f-4f57-9389-9675edda16f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:07.553744 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915289f6-6f0f-4f57-9389-9675edda16f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:07.554018 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915289f6-6f0f-4f57-9389-9675edda16f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:07.554735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915289f6-6f0f-4f57-9389-9675edda16f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:07.556833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915289f6-6f0f-4f57-9389-9675edda16f8 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:07.609473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 816/1594] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:07 2022] POST /identity/v3/users => generated 292 bytes in 316 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:07.610902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915289f6-6f0f-4f57-9389-9675edda16f8 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:07.632371 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915289f6-6f0f-4f57-9389-9675edda16f8 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:07.646379 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2592475d-18c4-4218-b85d-f525a2fd9242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:07.648177 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2592475d-18c4-4218-b85d-f525a2fd9242 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:07.649996 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2592475d-18c4-4218-b85d-f525a2fd9242 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:07.650319 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2592475d-18c4-4218-b85d-f525a2fd9242 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:07.651030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2592475d-18c4-4218-b85d-f525a2fd9242 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:07.656616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2592475d-18c4-4218-b85d-f525a2fd9242 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:07.663128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d559ef9-c6f2-4d48-9208-b993a139be38 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:07.677965 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f068990-d9ba-4aed-b352-06be1ee9d02b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:07.679041 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f068990-d9ba-4aed-b352-06be1ee9d02b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:07.679390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f068990-d9ba-4aed-b352-06be1ee9d02b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:07.679676 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f068990-d9ba-4aed-b352-06be1ee9d02b admin admin] PATH_INFO: `/v3/projects/f88dc43c613e45aab8456768d988fb0c` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:07.681235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f068990-d9ba-4aed-b352-06be1ee9d02b admin admin] RBAC: Authorizing `identity:delete_project(project_id=f88dc43c613e45aab8456768d988fb0c)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:07.684430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f068990-d9ba-4aed-b352-06be1ee9d02b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:07.691899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 817/1595] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:07 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:07.695196 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2592475d-18c4-4218-b85d-f525a2fd9242 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:07.718271 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2592475d-18c4-4218-b85d-f525a2fd9242 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:07.742632 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea3f09b-87e2-430b-9942-44f1fa95c3b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:07.746290 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea3f09b-87e2-430b-9942-44f1fa95c3b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:07.746290 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea3f09b-87e2-430b-9942-44f1fa95c3b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:07.746290 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea3f09b-87e2-430b-9942-44f1fa95c3b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:07.750496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea3f09b-87e2-430b-9942-44f1fa95c3b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:07.754014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea3f09b-87e2-430b-9942-44f1fa95c3b3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:07.768163 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f068990-d9ba-4aed-b352-06be1ee9d02b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f88dc43c613e45aab8456768d988fb0c', 'request_id': 'req-6f068990-d9ba-4aed-b352-06be1ee9d02b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:07.787207 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f068990-d9ba-4aed-b352-06be1ee9d02b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f88dc43c613e45aab8456768d988fb0c', 'request_id': 'req-6f068990-d9ba-4aed-b352-06be1ee9d02b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:07.792283 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 818/1596] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:07 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:07.794421 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea3f09b-87e2-430b-9942-44f1fa95c3b3 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:07.799741 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-0ea3f09b-87e2-430b-9942-44f1fa95c3b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:07.803140 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f068990-d9ba-4aed-b352-06be1ee9d02b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f88dc43c613e45aab8456768d988fb0c', 'request_id': 'req-6f068990-d9ba-4aed-b352-06be1ee9d02b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:07.832298 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f068990-d9ba-4aed-b352-06be1ee9d02b admin admin] The token cache is being invalidate because project f88dc43c613e45aab8456768d988fb0c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:07.832732 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f068990-d9ba-4aed-b352-06be1ee9d02b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:07.833752 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6f068990-d9ba-4aed-b352-06be1ee9d02b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:07.835969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 779/1597] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:07 2022] DELETE /identity/v3/projects/f88dc43c613e45aab8456768d988fb0c => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:58:07.837203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f068990-d9ba-4aed-b352-06be1ee9d02b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:07.841644 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6f068990-d9ba-4aed-b352-06be1ee9d02b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:07.845169 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea3f09b-87e2-430b-9942-44f1fa95c3b3 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:07.848800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:07.981392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f068990-d9ba-4aed-b352-06be1ee9d02b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:07.998572 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3824a1-6a35-4a54-a842-d5fec974672c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:07.999494 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3824a1-6a35-4a54-a842-d5fec974672c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:07.999975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3824a1-6a35-4a54-a842-d5fec974672c admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:08.000520 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3824a1-6a35-4a54-a842-d5fec974672c admin admin] PATH_INFO: `/v3/projects/ab3e02221593480a8b183f9cb050dd38` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:08.002042 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3824a1-6a35-4a54-a842-d5fec974672c admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab3e02221593480a8b183f9cb050dd38)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:08.004258 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3824a1-6a35-4a54-a842-d5fec974672c admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:08.013370 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cd4d10-2866-41ed-bf0c-ae7dcab5183d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:08.014318 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cd4d10-2866-41ed-bf0c-ae7dcab5183d admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:08.014715 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cd4d10-2866-41ed-bf0c-ae7dcab5183d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:08.015128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cd4d10-2866-41ed-bf0c-ae7dcab5183d admin admin] PATH_INFO: `/v3/projects/e3a23c2edf614693949411584f0056b2/users/5e82c788383741a58921c5412138c186/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:08.062706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cd4d10-2866-41ed-bf0c-ae7dcab5183d admin admin] RBAC: Authorizing `identity:create_grant(project_id=e3a23c2edf614693949411584f0056b2, user_id=5e82c788383741a58921c5412138c186, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:08.068631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cd4d10-2866-41ed-bf0c-ae7dcab5183d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:08.070850 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4e3824a1-6a35-4a54-a842-d5fec974672c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab3e02221593480a8b183f9cb050dd38', 'request_id': 'req-4e3824a1-6a35-4a54-a842-d5fec974672c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:08.080477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4e3824a1-6a35-4a54-a842-d5fec974672c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab3e02221593480a8b183f9cb050dd38', 'request_id': 'req-4e3824a1-6a35-4a54-a842-d5fec974672c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:08.091895 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4e3824a1-6a35-4a54-a842-d5fec974672c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab3e02221593480a8b183f9cb050dd38', 'request_id': 'req-4e3824a1-6a35-4a54-a842-d5fec974672c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:08.096866 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 780/1598] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:07 2022] PUT /identity/v3/projects/e3a23c2edf614693949411584f0056b2/users/5e82c788383741a58921c5412138c186/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:08.098323 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cdf688-c453-4345-ae23-623c594eb9ec None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:08.099053 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cdf688-c453-4345-ae23-623c594eb9ec None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:08.099915 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cdf688-c453-4345-ae23-623c594eb9ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:08.120491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4e3824a1-6a35-4a54-a842-d5fec974672c admin admin] The token cache is being invalidate because project ab3e02221593480a8b183f9cb050dd38 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:08.120491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4e3824a1-6a35-4a54-a842-d5fec974672c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:08.124056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4e3824a1-6a35-4a54-a842-d5fec974672c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:08.130474 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 819/1599] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:07 2022] DELETE /identity/v3/projects/ab3e02221593480a8b183f9cb050dd38 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:08.132140 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17c2c9e-0799-42ac-ad2e-baf5e97aed04 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:08.132490 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17c2c9e-0799-42ac-ad2e-baf5e97aed04 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:08.132911 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17c2c9e-0799-42ac-ad2e-baf5e97aed04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:08.192505 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-08cdf688-c453-4345-ae23-623c594eb9ec None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:08.193778 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-08cdf688-c453-4345-ae23-623c594eb9ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:08.221720 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-d17c2c9e-0799-42ac-ad2e-baf5e97aed04 None None] MFA Rules not processed for user `5e82c788383741a58921c5412138c186`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:08.224930 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d17c2c9e-0799-42ac-ad2e-baf5e97aed04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:08.245116 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:08.395302 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 781/1600] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 298 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:08.397002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b374839-7e9f-412a-a532-a84b9179ca07 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:08.398031 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b374839-7e9f-412a-a532-a84b9179ca07 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:08.399037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b374839-7e9f-412a-a532-a84b9179ca07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:08.420753 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 820/1601] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:08 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 290 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:08.422133 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c042588-31d6-4ad5-af94-2bf03588860b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:08.422473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c042588-31d6-4ad5-af94-2bf03588860b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:08.422807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c042588-31d6-4ad5-af94-2bf03588860b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:08.425434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4b374839-7e9f-412a-a532-a84b9179ca07 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:08.427304 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4b374839-7e9f-412a-a532-a84b9179ca07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:08.454889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 782/1602] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:08.456152 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b374839-7e9f-412a-a532-a84b9179ca07 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:08.470534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-5c042588-31d6-4ad5-af94-2bf03588860b None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:08.470534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-5c042588-31d6-4ad5-af94-2bf03588860b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:08.470848 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4b374839-7e9f-412a-a532-a84b9179ca07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:08.475950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b374839-7e9f-412a-a532-a84b9179ca07 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:08.480269 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 821/1603] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:08.484871 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddee7edd-566d-4005-a37a-a7c20dc2bb2c None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:08.485115 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb4db31-c6c8-4364-a1c8-9b56753bdd98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:08.485230 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddee7edd-566d-4005-a37a-a7c20dc2bb2c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:08.485416 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddee7edd-566d-4005-a37a-a7c20dc2bb2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:08.486094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb4db31-c6c8-4364-a1c8-9b56753bdd98 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:08.487111 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb4db31-c6c8-4364-a1c8-9b56753bdd98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:08.487602 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb4db31-c6c8-4364-a1c8-9b56753bdd98 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:08.488581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb4db31-c6c8-4364-a1c8-9b56753bdd98 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:08.491292 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb4db31-c6c8-4364-a1c8-9b56753bdd98 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:08.515099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-ddee7edd-566d-4005-a37a-a7c20dc2bb2c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:08.516383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-ddee7edd-566d-4005-a37a-a7c20dc2bb2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:08.532228 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 822/1604] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:08.532228 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddee7edd-566d-4005-a37a-a7c20dc2bb2c None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:08.532859 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-ddee7edd-566d-4005-a37a-a7c20dc2bb2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:08.546906 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 783/1605] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:08 2022] POST /identity/v3/projects => generated 381 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:08.546906 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb4db31-c6c8-4364-a1c8-9b56753bdd98 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:08.634125 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb4db31-c6c8-4364-a1c8-9b56753bdd98 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:08.640316 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702ef340-8319-4cf6-acf6-34024d3422ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:08.641318 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702ef340-8319-4cf6-acf6-34024d3422ec admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:08.641612 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702ef340-8319-4cf6-acf6-34024d3422ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:08.641896 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702ef340-8319-4cf6-acf6-34024d3422ec admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:08.642953 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702ef340-8319-4cf6-acf6-34024d3422ec admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:58:08.645549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702ef340-8319-4cf6-acf6-34024d3422ec admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:08.650632 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702ef340-8319-4cf6-acf6-34024d3422ec admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:08.653120 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:08.658588 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddee7edd-566d-4005-a37a-a7c20dc2bb2c None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:08.672870 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 784/1606] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:58:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:58:08.675609 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702ef340-8319-4cf6-acf6-34024d3422ec admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:08.693294 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702ef340-8319-4cf6-acf6-34024d3422ec admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:08.702059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed63555-6e32-4138-9fc3-a85d8d34d0f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:08.702797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed63555-6e32-4138-9fc3-a85d8d34d0f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:08.703223 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed63555-6e32-4138-9fc3-a85d8d34d0f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:08.703603 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed63555-6e32-4138-9fc3-a85d8d34d0f9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:08.704374 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed63555-6e32-4138-9fc3-a85d8d34d0f9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:58:08.705052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed63555-6e32-4138-9fc3-a85d8d34d0f9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:08.707634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed63555-6e32-4138-9fc3-a85d8d34d0f9 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:08.721638 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f190ea-96f6-4d4c-bd98-ae34e3daffcb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:08.722546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f190ea-96f6-4d4c-bd98-ae34e3daffcb service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:08.722827 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f190ea-96f6-4d4c-bd98-ae34e3daffcb service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:08.723129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f190ea-96f6-4d4c-bd98-ae34e3daffcb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:08.724962 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c6f190ea-96f6-4d4c-bd98-ae34e3daffcb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:08.727480 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 785/1607] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:58:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:58:08.729938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed63555-6e32-4138-9fc3-a85d8d34d0f9 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:08.752403 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed63555-6e32-4138-9fc3-a85d8d34d0f9 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:08.759623 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf3c508-51e5-4e3d-8bba-cd859c9228bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:08.760256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf3c508-51e5-4e3d-8bba-cd859c9228bd admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:08.760540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf3c508-51e5-4e3d-8bba-cd859c9228bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:08.760843 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf3c508-51e5-4e3d-8bba-cd859c9228bd admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:08.762461 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf3c508-51e5-4e3d-8bba-cd859c9228bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:08.764501 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf3c508-51e5-4e3d-8bba-cd859c9228bd admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:08.792437 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 786/1608] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:08 2022] POST /identity/v3/users => generated 297 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:08.792437 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf3c508-51e5-4e3d-8bba-cd859c9228bd admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:08.803453 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf3c508-51e5-4e3d-8bba-cd859c9228bd admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:08.805608 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f190ea-96f6-4d4c-bd98-ae34e3daffcb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:08.807441 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f190ea-96f6-4d4c-bd98-ae34e3daffcb service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:08.808475 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287e854d-c935-4755-b13b-6c4945e74d50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:08.810835 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287e854d-c935-4755-b13b-6c4945e74d50 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:08.811184 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287e854d-c935-4755-b13b-6c4945e74d50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:08.811461 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287e854d-c935-4755-b13b-6c4945e74d50 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:08.812187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287e854d-c935-4755-b13b-6c4945e74d50 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:08.815254 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287e854d-c935-4755-b13b-6c4945e74d50 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:08.823246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 823/1609] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:08 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:08.824891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f190ea-96f6-4d4c-bd98-ae34e3daffcb service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:08.834643 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:08.841317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f190ea-96f6-4d4c-bd98-ae34e3daffcb service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:08.848481 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d197bc67-fd55-4217-a814-bd66b281811c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:08.849071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d197bc67-fd55-4217-a814-bd66b281811c admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:08.849350 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d197bc67-fd55-4217-a814-bd66b281811c admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:08.849617 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d197bc67-fd55-4217-a814-bd66b281811c admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:08.850252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d197bc67-fd55-4217-a814-bd66b281811c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:08.852557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d197bc67-fd55-4217-a814-bd66b281811c admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:08.877507 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 787/1610] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:08 2022] POST /identity/v3/projects => generated 373 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:08.878361 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287e854d-c935-4755-b13b-6c4945e74d50 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:08.947126 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 824/1611] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:08 2022] POST /identity/v3/projects => generated 373 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:08.948143 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d197bc67-fd55-4217-a814-bd66b281811c admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:08.985673 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287e854d-c935-4755-b13b-6c4945e74d50 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:09.077334 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d197bc67-fd55-4217-a814-bd66b281811c admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:09.078826 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc4b850-7445-4e14-9edc-f583beb1427d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:09.079760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc4b850-7445-4e14-9edc-f583beb1427d admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:09.080175 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc4b850-7445-4e14-9edc-f583beb1427d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:09.080698 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc4b850-7445-4e14-9edc-f583beb1427d admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:09.081426 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc4b850-7445-4e14-9edc-f583beb1427d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:09.083688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc4b850-7445-4e14-9edc-f583beb1427d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:09.086127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b9754a-7b12-4a05-bf0c-90cc45c7070b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:09.086921 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b9754a-7b12-4a05-bf0c-90cc45c7070b admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:09.087222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b9754a-7b12-4a05-bf0c-90cc45c7070b admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:09.087502 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b9754a-7b12-4a05-bf0c-90cc45c7070b admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:09.088402 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b9754a-7b12-4a05-bf0c-90cc45c7070b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:09.091874 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b9754a-7b12-4a05-bf0c-90cc45c7070b admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:09.103417 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 788/1612] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:08 2022] GET /identity/v3/roles => generated 1420 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:09.104367 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc4b850-7445-4e14-9edc-f583beb1427d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:09.116914 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc4b850-7445-4e14-9edc-f583beb1427d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:09.122346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 825/1613] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:08 2022] POST /identity/v3/users => generated 287 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:09.125744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657f45aa-3a2c-426c-8c8c-076c2fb6359d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:09.125744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657f45aa-3a2c-426c-8c8c-076c2fb6359d admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:09.126003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b9754a-7b12-4a05-bf0c-90cc45c7070b admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:09.126114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657f45aa-3a2c-426c-8c8c-076c2fb6359d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:09.126114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657f45aa-3a2c-426c-8c8c-076c2fb6359d admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:09.126114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657f45aa-3a2c-426c-8c8c-076c2fb6359d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:09.127594 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657f45aa-3a2c-426c-8c8c-076c2fb6359d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:09.139613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b9754a-7b12-4a05-bf0c-90cc45c7070b admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:09.146072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cf48d3-1f2c-4dbc-83d8-15e8ef2e98bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:09.146953 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cf48d3-1f2c-4dbc-83d8-15e8ef2e98bc admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:09.147457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cf48d3-1f2c-4dbc-83d8-15e8ef2e98bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:09.147764 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cf48d3-1f2c-4dbc-83d8-15e8ef2e98bc admin admin] PATH_INFO: `/v3/projects/75b4f48624b84658bf50d2c95e001e3a/users/f2a1c22c27f14f7a8d0a7be7265d7b14/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:09.160443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 789/1614] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:09 2022] POST /identity/v3/users => generated 287 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:09.161413 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657f45aa-3a2c-426c-8c8c-076c2fb6359d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:09.167931 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79cf48d3-1f2c-4dbc-83d8-15e8ef2e98bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=75b4f48624b84658bf50d2c95e001e3a, user_id=f2a1c22c27f14f7a8d0a7be7265d7b14, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:09.171342 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79cf48d3-1f2c-4dbc-83d8-15e8ef2e98bc admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:09.172649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657f45aa-3a2c-426c-8c8c-076c2fb6359d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:09.179312 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443f4f65-a4e3-4852-96c3-8f0bf88e1e69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:09.179902 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443f4f65-a4e3-4852-96c3-8f0bf88e1e69 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:09.180204 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443f4f65-a4e3-4852-96c3-8f0bf88e1e69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:09.180456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443f4f65-a4e3-4852-96c3-8f0bf88e1e69 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:09.181103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443f4f65-a4e3-4852-96c3-8f0bf88e1e69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:09.182747 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443f4f65-a4e3-4852-96c3-8f0bf88e1e69 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:09.196146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 790/1615] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:09 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:09.197680 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443f4f65-a4e3-4852-96c3-8f0bf88e1e69 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:09.206101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 826/1616] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:09 2022] PUT /identity/v3/projects/75b4f48624b84658bf50d2c95e001e3a/users/f2a1c22c27f14f7a8d0a7be7265d7b14/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:09.207105 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cf48d3-1f2c-4dbc-83d8-15e8ef2e98bc admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:09.228777 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443f4f65-a4e3-4852-96c3-8f0bf88e1e69 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:09.256165 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:09.301839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cf48d3-1f2c-4dbc-83d8-15e8ef2e98bc admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:09.311887 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f877d5e7-47ae-4724-9246-1b7b0de6dbc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:09.313412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f877d5e7-47ae-4724-9246-1b7b0de6dbc7 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:09.314207 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f877d5e7-47ae-4724-9246-1b7b0de6dbc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:09.314687 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f877d5e7-47ae-4724-9246-1b7b0de6dbc7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:09.316597 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f877d5e7-47ae-4724-9246-1b7b0de6dbc7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:09.321649 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f877d5e7-47ae-4724-9246-1b7b0de6dbc7 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:09.340086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 827/1617] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:09 2022] GET /identity/v3/roles => generated 1420 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:09.341402 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f877d5e7-47ae-4724-9246-1b7b0de6dbc7 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:09.354035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f877d5e7-47ae-4724-9246-1b7b0de6dbc7 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:09.362517 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9200ca6-c6ce-4c38-a587-fa05f7674923 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:09.363030 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cde361-bf37-4008-814d-dd520f74dd88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:09.363191 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9200ca6-c6ce-4c38-a587-fa05f7674923 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:09.363569 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9200ca6-c6ce-4c38-a587-fa05f7674923 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:09.363730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9200ca6-c6ce-4c38-a587-fa05f7674923 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:09.364520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cde361-bf37-4008-814d-dd520f74dd88 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:09.364613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9200ca6-c6ce-4c38-a587-fa05f7674923 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:09.364955 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cde361-bf37-4008-814d-dd520f74dd88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:09.365322 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cde361-bf37-4008-814d-dd520f74dd88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:09.366026 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9200ca6-c6ce-4c38-a587-fa05f7674923 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:09.366297 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cde361-bf37-4008-814d-dd520f74dd88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:09.368448 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cde361-bf37-4008-814d-dd520f74dd88 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:09.381598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 828/1618] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:09 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:09.385566 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9200ca6-c6ce-4c38-a587-fa05f7674923 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:09.387507 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 791/1619] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:09 2022] GET /identity/v3/roles => generated 1420 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:09.388482 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cde361-bf37-4008-814d-dd520f74dd88 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:09.404578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9200ca6-c6ce-4c38-a587-fa05f7674923 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:09.404768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cde361-bf37-4008-814d-dd520f74dd88 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:09.409156 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffbf8764-a07b-4b09-bd64-3762ccc76e40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:09.409719 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c482082-147c-4427-805b-c31478928585 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:09.409867 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbf8764-a07b-4b09-bd64-3762ccc76e40 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:09.410149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbf8764-a07b-4b09-bd64-3762ccc76e40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:09.410404 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c482082-147c-4427-805b-c31478928585 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:09.410519 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbf8764-a07b-4b09-bd64-3762ccc76e40 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:09.410685 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c482082-147c-4427-805b-c31478928585 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:09.410986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c482082-147c-4427-805b-c31478928585 admin admin] PATH_INFO: `/v3/projects/a850973394834b1c9a2eb5370e303a21/users/3fad6c6ca4614270856c689c5bb46a5f/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:09.411103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffbf8764-a07b-4b09-bd64-3762ccc76e40 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:09.414093 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffbf8764-a07b-4b09-bd64-3762ccc76e40 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:09.436055 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 792/1620] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:09 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:09.436962 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffbf8764-a07b-4b09-bd64-3762ccc76e40 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:09.442224 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c482082-147c-4427-805b-c31478928585 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a850973394834b1c9a2eb5370e303a21, user_id=3fad6c6ca4614270856c689c5bb46a5f, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:09.446851 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c482082-147c-4427-805b-c31478928585 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:09.456039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffbf8764-a07b-4b09-bd64-3762ccc76e40 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:09.467641 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 829/1621] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:09 2022] PUT /identity/v3/projects/a850973394834b1c9a2eb5370e303a21/users/3fad6c6ca4614270856c689c5bb46a5f/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:09.468802 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c482082-147c-4427-805b-c31478928585 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:09.481866 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:09.564824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c482082-147c-4427-805b-c31478928585 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:09.568600 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f70076b-83cb-46e0-8b4b-ad5a44a88713 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:09.572595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f70076b-83cb-46e0-8b4b-ad5a44a88713 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:09.572914 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f70076b-83cb-46e0-8b4b-ad5a44a88713 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:09.573203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f70076b-83cb-46e0-8b4b-ad5a44a88713 admin admin] PATH_INFO: `/v3/projects/75b4f48624b84658bf50d2c95e001e3a/users/f2a1c22c27f14f7a8d0a7be7265d7b14/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:09.586439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16fb8ec-d5fd-4ea3-b1d6-2aa01725bfde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:09.587196 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16fb8ec-d5fd-4ea3-b1d6-2aa01725bfde admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:09.587481 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16fb8ec-d5fd-4ea3-b1d6-2aa01725bfde admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:09.587758 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16fb8ec-d5fd-4ea3-b1d6-2aa01725bfde admin admin] PATH_INFO: `/v3/projects/43a1557bbe394619a8e02760d2dc12d3/users/4e9cc16126cc4dd1a21699dfb11dc7d7/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:09.595755 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f70076b-83cb-46e0-8b4b-ad5a44a88713 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75b4f48624b84658bf50d2c95e001e3a, user_id=f2a1c22c27f14f7a8d0a7be7265d7b14, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:09.601906 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f70076b-83cb-46e0-8b4b-ad5a44a88713 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:09.607818 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16fb8ec-d5fd-4ea3-b1d6-2aa01725bfde admin admin] RBAC: Authorizing `identity:create_grant(project_id=43a1557bbe394619a8e02760d2dc12d3, user_id=4e9cc16126cc4dd1a21699dfb11dc7d7, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:09.611513 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16fb8ec-d5fd-4ea3-b1d6-2aa01725bfde admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:09.615522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 793/1622] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:09 2022] PUT /identity/v3/projects/75b4f48624b84658bf50d2c95e001e3a/users/f2a1c22c27f14f7a8d0a7be7265d7b14/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:09.617124 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92df776-0522-4eb8-9f98-d545292e706c None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:09.617480 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92df776-0522-4eb8-9f98-d545292e706c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:09.617823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92df776-0522-4eb8-9f98-d545292e706c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:09.639471 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 830/1623] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:09 2022] PUT /identity/v3/projects/43a1557bbe394619a8e02760d2dc12d3/users/4e9cc16126cc4dd1a21699dfb11dc7d7/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:09.639666 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7811f3-ca43-4651-bc18-1e0ab921ff3e None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:09.639893 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7811f3-ca43-4651-bc18-1e0ab921ff3e None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:09.640178 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7811f3-ca43-4651-bc18-1e0ab921ff3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:09.656623 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b92df776-0522-4eb8-9f98-d545292e706c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:09.657545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b92df776-0522-4eb8-9f98-d545292e706c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:09.674773 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-df7811f3-ca43-4651-bc18-1e0ab921ff3e None None] MFA Rules not processed for user `f2a1c22c27f14f7a8d0a7be7265d7b14`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:09.676042 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-df7811f3-ca43-4651-bc18-1e0ab921ff3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:09.730931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 794/1624] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:09.731958 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a24d5c-517c-42d2-8d14-23a3f63147ae None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:09.732244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a24d5c-517c-42d2-8d14-23a3f63147ae None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:09.732510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a24d5c-517c-42d2-8d14-23a3f63147ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:09.755240 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 831/1625] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:09 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:09.756092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7811f3-ca43-4651-bc18-1e0ab921ff3e None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:09.757362 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-df7811f3-ca43-4651-bc18-1e0ab921ff3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:09.764159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-30a24d5c-517c-42d2-8d14-23a3f63147ae None None] MFA Rules not processed for user `4e9cc16126cc4dd1a21699dfb11dc7d7`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:09.766208 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-30a24d5c-517c-42d2-8d14-23a3f63147ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:09.804897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:09.848816 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7811f3-ca43-4651-bc18-1e0ab921ff3e None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:09.856568 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607cf59-bbc0-43dd-997b-8539526408c2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:09.857441 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607cf59-bbc0-43dd-997b-8539526408c2 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:09.857976 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607cf59-bbc0-43dd-997b-8539526408c2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:09.858515 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607cf59-bbc0-43dd-997b-8539526408c2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:09.862799 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 795/1626] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:09 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:09.863415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a24d5c-517c-42d2-8d14-23a3f63147ae None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:09.874971 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e607cf59-bbc0-43dd-997b-8539526408c2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:09.879277 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e607cf59-bbc0-43dd-997b-8539526408c2 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:09.897458 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 832/1627] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:09.898490 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607cf59-bbc0-43dd-997b-8539526408c2 service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:09.909686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607cf59-bbc0-43dd-997b-8539526408c2 service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:09.914807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a43c536-e693-44ee-9ca9-5d84958079f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:09.915478 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a43c536-e693-44ee-9ca9-5d84958079f4 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:09.915763 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a43c536-e693-44ee-9ca9-5d84958079f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:09.916039 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a43c536-e693-44ee-9ca9-5d84958079f4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:09.916684 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a43c536-e693-44ee-9ca9-5d84958079f4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:09.920270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a43c536-e693-44ee-9ca9-5d84958079f4 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:09.954210 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:09.963630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 833/1628] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:09 2022] POST /identity/v3/projects => generated 375 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:09.963745 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a24d5c-517c-42d2-8d14-23a3f63147ae None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:09.980319 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a43c536-e693-44ee-9ca9-5d84958079f4 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:10.035211 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe6b84e-fc87-4afb-b24e-2b413d3c4486 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:10.036022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe6b84e-fc87-4afb-b24e-2b413d3c4486 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:10.036304 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe6b84e-fc87-4afb-b24e-2b413d3c4486 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:10.036577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe6b84e-fc87-4afb-b24e-2b413d3c4486 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:10.124089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a43c536-e693-44ee-9ca9-5d84958079f4 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:10.134190 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf223c38-6d40-4eab-9aed-dd4269450ec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:10.135562 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf223c38-6d40-4eab-9aed-dd4269450ec0 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:10.138014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf223c38-6d40-4eab-9aed-dd4269450ec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:10.138296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf223c38-6d40-4eab-9aed-dd4269450ec0 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:10.139294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf223c38-6d40-4eab-9aed-dd4269450ec0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:10.144001 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf223c38-6d40-4eab-9aed-dd4269450ec0 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:10.175911 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe6b84e-fc87-4afb-b24e-2b413d3c4486 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:10.180094 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 834/1629] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:09 2022] POST /identity/v3/users => generated 294 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:10.180794 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe6b84e-fc87-4afb-b24e-2b413d3c4486 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:10.183541 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf223c38-6d40-4eab-9aed-dd4269450ec0 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:10.196397 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf223c38-6d40-4eab-9aed-dd4269450ec0 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:10.202361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caf8cd0-fd24-46a2-8d24-bc99e4fba569 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:10.202685 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caf8cd0-fd24-46a2-8d24-bc99e4fba569 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:10.203184 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caf8cd0-fd24-46a2-8d24-bc99e4fba569 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:10.203465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caf8cd0-fd24-46a2-8d24-bc99e4fba569 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:10.204126 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caf8cd0-fd24-46a2-8d24-bc99e4fba569 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:10.205834 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caf8cd0-fd24-46a2-8d24-bc99e4fba569 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:10.220690 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 796/1630] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:09 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:10.227446 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 835/1631] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:10 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:10.232657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe6b84e-fc87-4afb-b24e-2b413d3c4486 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:10.251227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe6b84e-fc87-4afb-b24e-2b413d3c4486 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:10.259938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fccfa71-1b59-438a-a494-a9ce7f52dcd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:10.259938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fccfa71-1b59-438a-a494-a9ce7f52dcd4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:10.260326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fccfa71-1b59-438a-a494-a9ce7f52dcd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:10.261734 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fccfa71-1b59-438a-a494-a9ce7f52dcd4 admin admin] PATH_INFO: `/v3/projects/e44d58ff3c704162be8230616968b2d6/users/92a073fcff47436f9a623f115b49759e/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:10.286707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fccfa71-1b59-438a-a494-a9ce7f52dcd4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e44d58ff3c704162be8230616968b2d6, user_id=92a073fcff47436f9a623f115b49759e, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:10.289791 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fccfa71-1b59-438a-a494-a9ce7f52dcd4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:10.331658 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 797/1632] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:10 2022] PUT /identity/v3/projects/e44d58ff3c704162be8230616968b2d6/users/92a073fcff47436f9a623f115b49759e/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:10.336035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caf8cd0-fd24-46a2-8d24-bc99e4fba569 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:10.356485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caf8cd0-fd24-46a2-8d24-bc99e4fba569 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:10.479467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f31085-01c9-4680-90ee-5a23cbf3c6be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:10.480196 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f31085-01c9-4680-90ee-5a23cbf3c6be admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:10.480983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f31085-01c9-4680-90ee-5a23cbf3c6be admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:10.481257 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f31085-01c9-4680-90ee-5a23cbf3c6be admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:10.481963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f31085-01c9-4680-90ee-5a23cbf3c6be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:10.483928 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f31085-01c9-4680-90ee-5a23cbf3c6be admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:10.504849 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 836/1633] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:10 2022] GET /identity/v3/roles => generated 1420 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:10.506001 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fccfa71-1b59-438a-a494-a9ce7f52dcd4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:10.508497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f31085-01c9-4680-90ee-5a23cbf3c6be admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:10.514760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9fccfa71-1b59-438a-a494-a9ce7f52dcd4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:10.531534 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:10.543791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f31085-01c9-4680-90ee-5a23cbf3c6be admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:10.561153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59513f39-7ec0-4174-beb8-b06db8703c5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:10.561845 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59513f39-7ec0-4174-beb8-b06db8703c5b admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:10.562511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59513f39-7ec0-4174-beb8-b06db8703c5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:10.562511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59513f39-7ec0-4174-beb8-b06db8703c5b admin admin] PATH_INFO: `/v3/projects/e44d58ff3c704162be8230616968b2d6/users/92a073fcff47436f9a623f115b49759e/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:10.571869 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59513f39-7ec0-4174-beb8-b06db8703c5b admin admin] RBAC: Authorizing `identity:create_grant(project_id=e44d58ff3c704162be8230616968b2d6, user_id=92a073fcff47436f9a623f115b49759e, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:10.577076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59513f39-7ec0-4174-beb8-b06db8703c5b admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:10.615900 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 837/1634] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:10 2022] PUT /identity/v3/projects/e44d58ff3c704162be8230616968b2d6/users/92a073fcff47436f9a623f115b49759e/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:58:10.617275 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fccfa71-1b59-438a-a494-a9ce7f52dcd4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:10.627950 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc848e79-0448-4be0-af07-313765c31d2e None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:10.628265 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc848e79-0448-4be0-af07-313765c31d2e None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:10.628728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc848e79-0448-4be0-af07-313765c31d2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:10.704048 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-dc848e79-0448-4be0-af07-313765c31d2e None None] MFA Rules not processed for user `92a073fcff47436f9a623f115b49759e`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:10.707003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-dc848e79-0448-4be0-af07-313765c31d2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:10.725545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8b6865-4baf-4487-8ee6-f6a72e86d15b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:10.728603 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8b6865-4baf-4487-8ee6-f6a72e86d15b service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:10.728932 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8b6865-4baf-4487-8ee6-f6a72e86d15b service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:10.729606 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8b6865-4baf-4487-8ee6-f6a72e86d15b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:10.834491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:10.837550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8b6865-4baf-4487-8ee6-f6a72e86d15b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:10.840369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8b6865-4baf-4487-8ee6-f6a72e86d15b service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:10.852033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 798/1635] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:10 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:58:10.885550 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 838/1636] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:10 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:58:10.894919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8b6865-4baf-4487-8ee6-f6a72e86d15b service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:10.905299 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8b6865-4baf-4487-8ee6-f6a72e86d15b service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:10.920329 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919f2691-7009-4260-b41d-d1ea7425fb01 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:10.920994 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919f2691-7009-4260-b41d-d1ea7425fb01 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:10.921222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919f2691-7009-4260-b41d-d1ea7425fb01 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:10.921937 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919f2691-7009-4260-b41d-d1ea7425fb01 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:11.018014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919f2691-7009-4260-b41d-d1ea7425fb01 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:11.019903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919f2691-7009-4260-b41d-d1ea7425fb01 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:11.033719 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 799/1637] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:10 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:11.688513 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63af03a-fd4e-44e2-ac75-f72060da04bf None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:11.689172 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63af03a-fd4e-44e2-ac75-f72060da04bf None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:11.689172 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63af03a-fd4e-44e2-ac75-f72060da04bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:11.748342 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-b63af03a-fd4e-44e2-ac75-f72060da04bf None None] MFA Rules not processed for user `3fad6c6ca4614270856c689c5bb46a5f`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:11.751031 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b63af03a-fd4e-44e2-ac75-f72060da04bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:11.836408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 839/1638] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:11 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:11.847394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919f2691-7009-4260-b41d-d1ea7425fb01 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:11.888108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:11.986797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919f2691-7009-4260-b41d-d1ea7425fb01 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:11.993372 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c9519e-e157-4b77-8c75-d263609e4091 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:11.994155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c9519e-e157-4b77-8c75-d263609e4091 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:11.994504 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c9519e-e157-4b77-8c75-d263609e4091 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:11.994738 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c9519e-e157-4b77-8c75-d263609e4091 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:11.995424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c9519e-e157-4b77-8c75-d263609e4091 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:11.997585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c9519e-e157-4b77-8c75-d263609e4091 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:12.039822 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 800/1639] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:11 2022] POST /identity/v3/projects => generated 371 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:12.040843 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63af03a-fd4e-44e2-ac75-f72060da04bf None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:12.054564 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63af03a-fd4e-44e2-ac75-f72060da04bf None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:12.155462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6812fd9-8c56-4e30-b375-b02a0d55adac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:12.156241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6812fd9-8c56-4e30-b375-b02a0d55adac admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:12.156724 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6812fd9-8c56-4e30-b375-b02a0d55adac admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:12.157243 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6812fd9-8c56-4e30-b375-b02a0d55adac admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:12.158220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6812fd9-8c56-4e30-b375-b02a0d55adac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:12.160955 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6812fd9-8c56-4e30-b375-b02a0d55adac admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:12.192744 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 840/1640] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:12 2022] POST /identity/v3/users => generated 292 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:12.196353 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c9519e-e157-4b77-8c75-d263609e4091 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:12.209378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c9519e-e157-4b77-8c75-d263609e4091 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:12.216184 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1945a06a-ead8-48f1-be08-f783e6264e92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:12.216912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1945a06a-ead8-48f1-be08-f783e6264e92 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:12.217259 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1945a06a-ead8-48f1-be08-f783e6264e92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:12.217595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1945a06a-ead8-48f1-be08-f783e6264e92 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:12.218521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1945a06a-ead8-48f1-be08-f783e6264e92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:12.220335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1945a06a-ead8-48f1-be08-f783e6264e92 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:12.387024 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 801/1641] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:12 2022] GET /identity/v3/roles => generated 1420 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:12.388470 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6812fd9-8c56-4e30-b375-b02a0d55adac admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:12.404075 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6812fd9-8c56-4e30-b375-b02a0d55adac admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:12.412509 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b893bf15-80ee-461d-ac07-af1fba2d921f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:12.413489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b893bf15-80ee-461d-ac07-af1fba2d921f admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:12.413773 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b893bf15-80ee-461d-ac07-af1fba2d921f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:12.414106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b893bf15-80ee-461d-ac07-af1fba2d921f admin admin] PATH_INFO: `/v3/projects/94e12a99f9f541e5a3b2b643ab0c0332/users/201e23b6153b41c5b928747f417c0774/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:12.446225 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b893bf15-80ee-461d-ac07-af1fba2d921f admin admin] RBAC: Authorizing `identity:create_grant(project_id=94e12a99f9f541e5a3b2b643ab0c0332, user_id=201e23b6153b41c5b928747f417c0774, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:12.450431 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b893bf15-80ee-461d-ac07-af1fba2d921f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:12.470326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 841/1642] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:12 2022] PUT /identity/v3/projects/94e12a99f9f541e5a3b2b643ab0c0332/users/201e23b6153b41c5b928747f417c0774/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:12.474944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1945a06a-ead8-48f1-be08-f783e6264e92 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:12.493201 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1945a06a-ead8-48f1-be08-f783e6264e92 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:12.557835 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffe88a9-0dea-45a0-9751-8284466aa2f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:12.558809 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffe88a9-0dea-45a0-9751-8284466aa2f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:12.559276 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffe88a9-0dea-45a0-9751-8284466aa2f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:12.559556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffe88a9-0dea-45a0-9751-8284466aa2f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:12.560212 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffe88a9-0dea-45a0-9751-8284466aa2f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:12.561823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffe88a9-0dea-45a0-9751-8284466aa2f0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:12.572667 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 802/1643] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:12 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:12.577828 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b893bf15-80ee-461d-ac07-af1fba2d921f admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:12.591177 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b893bf15-80ee-461d-ac07-af1fba2d921f admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:12.595955 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0b2142-1061-48c0-af73-479a69cfdf07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:12.596632 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0b2142-1061-48c0-af73-479a69cfdf07 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:12.597326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0b2142-1061-48c0-af73-479a69cfdf07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:12.597326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0b2142-1061-48c0-af73-479a69cfdf07 admin admin] PATH_INFO: `/v3/projects/94e12a99f9f541e5a3b2b643ab0c0332/users/201e23b6153b41c5b928747f417c0774/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:12.599148 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0b2142-1061-48c0-af73-479a69cfdf07 admin admin] RBAC: Authorizing `identity:create_grant(project_id=94e12a99f9f541e5a3b2b643ab0c0332, user_id=201e23b6153b41c5b928747f417c0774, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:12.602171 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0b2142-1061-48c0-af73-479a69cfdf07 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:12.618491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 842/1644] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:12 2022] PUT /identity/v3/projects/94e12a99f9f541e5a3b2b643ab0c0332/users/201e23b6153b41c5b928747f417c0774/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:13.710074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84eda2fd-b9c3-46d3-ba06-ca4390ae3b43 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:13.710922 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84eda2fd-b9c3-46d3-ba06-ca4390ae3b43 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:13.711426 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84eda2fd-b9c3-46d3-ba06-ca4390ae3b43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:13.753461 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0b2142-1061-48c0-af73-479a69cfdf07 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:13.784853 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-84eda2fd-b9c3-46d3-ba06-ca4390ae3b43 None None] MFA Rules not processed for user `201e23b6153b41c5b928747f417c0774`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:13.786535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-84eda2fd-b9c3-46d3-ba06-ca4390ae3b43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:13.833881 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:13.889265 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0b2142-1061-48c0-af73-479a69cfdf07 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:13.900087 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca6461c-dc1d-451d-950f-2ddb2bf3f1e6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:13.902498 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca6461c-dc1d-451d-950f-2ddb2bf3f1e6 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:13.903067 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca6461c-dc1d-451d-950f-2ddb2bf3f1e6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:13.905276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca6461c-dc1d-451d-950f-2ddb2bf3f1e6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:13.908630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-9ca6461c-dc1d-451d-950f-2ddb2bf3f1e6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:13.965914 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 803/1645] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:13 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:13.967821 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84eda2fd-b9c3-46d3-ba06-ca4390ae3b43 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:14.005250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:14.159150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca6461c-dc1d-451d-950f-2ddb2bf3f1e6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:14.161082 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca6461c-dc1d-451d-950f-2ddb2bf3f1e6 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:14.192005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 843/1646] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:13 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:14.207424 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca6461c-dc1d-451d-950f-2ddb2bf3f1e6 service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:14.247891 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84eda2fd-b9c3-46d3-ba06-ca4390ae3b43 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:14.269698 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad123bc1-c2c0-4edd-87d6-3758c25fedc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:14.270844 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad123bc1-c2c0-4edd-87d6-3758c25fedc6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:14.277753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad123bc1-c2c0-4edd-87d6-3758c25fedc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:14.278506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad123bc1-c2c0-4edd-87d6-3758c25fedc6 admin admin] PATH_INFO: `/v3/users/5e82c788383741a58921c5412138c186` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:14.284706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad123bc1-c2c0-4edd-87d6-3758c25fedc6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e82c788383741a58921c5412138c186)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:14.290591 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad123bc1-c2c0-4edd-87d6-3758c25fedc6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:14.393979 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca6461c-dc1d-451d-950f-2ddb2bf3f1e6 service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:14.405778 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7db742-4771-4ce9-b386-90abb662693c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:14.406395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7db742-4771-4ce9-b386-90abb662693c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:14.406678 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7db742-4771-4ce9-b386-90abb662693c admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:14.407008 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7db742-4771-4ce9-b386-90abb662693c admin admin] PATH_INFO: `/v3/users/4e9cc16126cc4dd1a21699dfb11dc7d7` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:14.408348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7db742-4771-4ce9-b386-90abb662693c admin admin] RBAC: Authorizing `identity:delete_user(user_id=4e9cc16126cc4dd1a21699dfb11dc7d7)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:14.410495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7db742-4771-4ce9-b386-90abb662693c admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:14.434262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ad123bc1-c2c0-4edd-87d6-3758c25fedc6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e82c788383741a58921c5412138c186'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:14.435199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ad123bc1-c2c0-4edd-87d6-3758c25fedc6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e82c788383741a58921c5412138c186'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:14.459844 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ad123bc1-c2c0-4edd-87d6-3758c25fedc6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e82c788383741a58921c5412138c186'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:14.460880 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ad123bc1-c2c0-4edd-87d6-3758c25fedc6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e82c788383741a58921c5412138c186'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:14.493537 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ad123bc1-c2c0-4edd-87d6-3758c25fedc6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e82c788383741a58921c5412138c186'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:14.512790 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 804/1647] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:13 2022] DELETE /identity/v3/users/5e82c788383741a58921c5412138c186 => generated 0 bytes in 546 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:14.514371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad123bc1-c2c0-4edd-87d6-3758c25fedc6 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:14.516448 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ad123bc1-c2c0-4edd-87d6-3758c25fedc6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:14.524550 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-fa7db742-4771-4ce9-b386-90abb662693c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4e9cc16126cc4dd1a21699dfb11dc7d7'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:14.525273 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-fa7db742-4771-4ce9-b386-90abb662693c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4e9cc16126cc4dd1a21699dfb11dc7d7'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:14.543307 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-fa7db742-4771-4ce9-b386-90abb662693c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4e9cc16126cc4dd1a21699dfb11dc7d7'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:14.568033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-fa7db742-4771-4ce9-b386-90abb662693c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4e9cc16126cc4dd1a21699dfb11dc7d7'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:14.577581 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-fa7db742-4771-4ce9-b386-90abb662693c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4e9cc16126cc4dd1a21699dfb11dc7d7'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:14.581968 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:14.587104 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 844/1648] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:14 2022] DELETE /identity/v3/users/4e9cc16126cc4dd1a21699dfb11dc7d7 => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:14.588531 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bcd989-9b95-4933-a4fe-9be593b99941 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:14.588817 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bcd989-9b95-4933-a4fe-9be593b99941 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:14.589091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bcd989-9b95-4933-a4fe-9be593b99941 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:14.625278 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-49bcd989-9b95-4933-a4fe-9be593b99941 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:14.626461 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-49bcd989-9b95-4933-a4fe-9be593b99941 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:14.657170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad123bc1-c2c0-4edd-87d6-3758c25fedc6 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:14.747751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:14.753549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af07564-72f1-4994-86a8-7dcb7c24d769 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:14.754428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af07564-72f1-4994-86a8-7dcb7c24d769 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:14.754811 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af07564-72f1-4994-86a8-7dcb7c24d769 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:14.755200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af07564-72f1-4994-86a8-7dcb7c24d769 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:14.758744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8af07564-72f1-4994-86a8-7dcb7c24d769 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:14.813347 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 845/1649] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:14.817928 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6164f1-dfa5-4732-9cd9-f0a2ea14005b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:14.819292 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6164f1-dfa5-4732-9cd9-f0a2ea14005b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:14.819595 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6164f1-dfa5-4732-9cd9-f0a2ea14005b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:14.855171 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af07564-72f1-4994-86a8-7dcb7c24d769 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:14.855171 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af07564-72f1-4994-86a8-7dcb7c24d769 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:14.868200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 805/1650] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:14 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:14.869330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af07564-72f1-4994-86a8-7dcb7c24d769 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:14.871200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8af07564-72f1-4994-86a8-7dcb7c24d769 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:14.883274 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-1e6164f1-dfa5-4732-9cd9-f0a2ea14005b None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:14.887034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-1e6164f1-dfa5-4732-9cd9-f0a2ea14005b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:14.929730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 846/1651] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:14.977053 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af07564-72f1-4994-86a8-7dcb7c24d769 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:14.979092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6164f1-dfa5-4732-9cd9-f0a2ea14005b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:14.981100 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-1e6164f1-dfa5-4732-9cd9-f0a2ea14005b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:14.997579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5b79fb-8bd8-43c4-9ba1-25aeedd50ae4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:14.999060 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5b79fb-8bd8-43c4-9ba1-25aeedd50ae4 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:14.999628 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5b79fb-8bd8-43c4-9ba1-25aeedd50ae4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:15.000068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5b79fb-8bd8-43c4-9ba1-25aeedd50ae4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:15.013188 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6164f1-dfa5-4732-9cd9-f0a2ea14005b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:15.024427 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5b79fb-8bd8-43c4-9ba1-25aeedd50ae4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:15.028145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5b79fb-8bd8-43c4-9ba1-25aeedd50ae4 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:15.041337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71aea933-b864-4cb9-bedc-de9434440e4b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:15.042056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71aea933-b864-4cb9-bedc-de9434440e4b service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:15.042373 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71aea933-b864-4cb9-bedc-de9434440e4b service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:15.042682 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71aea933-b864-4cb9-bedc-de9434440e4b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:15.049866 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 806/1652] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:15.106483 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71aea933-b864-4cb9-bedc-de9434440e4b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:15.111847 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71aea933-b864-4cb9-bedc-de9434440e4b service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:15.139338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 847/1653] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:58:16.766641 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5b79fb-8bd8-43c4-9ba1-25aeedd50ae4 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:16.768429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5d5b79fb-8bd8-43c4-9ba1-25aeedd50ae4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:16.783080 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5b79fb-8bd8-43c4-9ba1-25aeedd50ae4 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:16.787185 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabb1b96-8d64-495f-97c4-97849b4e1b9d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:16.787895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabb1b96-8d64-495f-97c4-97849b4e1b9d service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:16.788174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabb1b96-8d64-495f-97c4-97849b4e1b9d service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:16.788523 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aabb1b96-8d64-495f-97c4-97849b4e1b9d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:16.791304 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-aabb1b96-8d64-495f-97c4-97849b4e1b9d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:16.895721 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71aea933-b864-4cb9-bedc-de9434440e4b service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:16.897632 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-71aea933-b864-4cb9-bedc-de9434440e4b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:16.926792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71aea933-b864-4cb9-bedc-de9434440e4b service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:16.962178 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995ff8d9-83de-4461-ab9c-1426767c3277 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:16.963456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995ff8d9-83de-4461-ab9c-1426767c3277 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:16.969951 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995ff8d9-83de-4461-ab9c-1426767c3277 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:16.972675 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995ff8d9-83de-4461-ab9c-1426767c3277 admin admin] PATH_INFO: `/v3/projects/43a1557bbe394619a8e02760d2dc12d3` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:16.983867 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995ff8d9-83de-4461-ab9c-1426767c3277 admin admin] RBAC: Authorizing `identity:delete_project(project_id=43a1557bbe394619a8e02760d2dc12d3)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:16.994023 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995ff8d9-83de-4461-ab9c-1426767c3277 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:17.023479 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabb1b96-8d64-495f-97c4-97849b4e1b9d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:17.028799 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aabb1b96-8d64-495f-97c4-97849b4e1b9d service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:17.059596 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 807/1654] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:16 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:17.061769 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabb1b96-8d64-495f-97c4-97849b4e1b9d service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:17.065135 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-aabb1b96-8d64-495f-97c4-97849b4e1b9d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:17.102351 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aabb1b96-8d64-495f-97c4-97849b4e1b9d service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:17.113646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:17.114599 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:17.115747 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:17.115747 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3 admin admin] PATH_INFO: `/v3/projects/e3a23c2edf614693949411584f0056b2` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:17.121486 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e3a23c2edf614693949411584f0056b2)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:17.125141 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:17.146405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-995ff8d9-83de-4461-ab9c-1426767c3277 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '43a1557bbe394619a8e02760d2dc12d3', 'request_id': 'req-995ff8d9-83de-4461-ab9c-1426767c3277'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.170532 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-995ff8d9-83de-4461-ab9c-1426767c3277 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '43a1557bbe394619a8e02760d2dc12d3', 'request_id': 'req-995ff8d9-83de-4461-ab9c-1426767c3277'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.180252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-995ff8d9-83de-4461-ab9c-1426767c3277 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '43a1557bbe394619a8e02760d2dc12d3', 'request_id': 'req-995ff8d9-83de-4461-ab9c-1426767c3277'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.205181 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:17.206332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-995ff8d9-83de-4461-ab9c-1426767c3277 admin admin] The token cache is being invalidate because project 43a1557bbe394619a8e02760d2dc12d3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:17.206759 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-995ff8d9-83de-4461-ab9c-1426767c3277 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.208801 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-995ff8d9-83de-4461-ab9c-1426767c3277 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.216908 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 848/1655] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:16 2022] DELETE /identity/v3/projects/43a1557bbe394619a8e02760d2dc12d3 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:17.218715 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995ff8d9-83de-4461-ab9c-1426767c3277 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:17.220967 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-995ff8d9-83de-4461-ab9c-1426767c3277 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:17.232789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e3a23c2edf614693949411584f0056b2', 'request_id': 'req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.248255 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e3a23c2edf614693949411584f0056b2', 'request_id': 'req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.261815 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e3a23c2edf614693949411584f0056b2', 'request_id': 'req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.279524 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3 admin admin] The token cache is being invalidate because project e3a23c2edf614693949411584f0056b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:17.280293 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.281393 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.286497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 808/1656] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:17 2022] DELETE /identity/v3/projects/e3a23c2edf614693949411584f0056b2 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:17.288271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:17.290101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:17.363391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995ff8d9-83de-4461-ab9c-1426767c3277 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:17.382896 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:17.445190 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ada425-90e5-4c4f-b0d1-5e8fa6be7cf3 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:17.459844 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424e09bc-f768-42e7-84f7-cd0c5a7d78b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:17.460521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424e09bc-f768-42e7-84f7-cd0c5a7d78b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:17.460810 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424e09bc-f768-42e7-84f7-cd0c5a7d78b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:17.461380 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424e09bc-f768-42e7-84f7-cd0c5a7d78b7 admin admin] PATH_INFO: `/v3/users/f2a1c22c27f14f7a8d0a7be7265d7b14` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:17.462992 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424e09bc-f768-42e7-84f7-cd0c5a7d78b7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f2a1c22c27f14f7a8d0a7be7265d7b14)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:17.466607 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424e09bc-f768-42e7-84f7-cd0c5a7d78b7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:17.492563 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14344471-8829-4843-8fcc-1885edaf4530 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:17.494384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14344471-8829-4843-8fcc-1885edaf4530 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:17.494672 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14344471-8829-4843-8fcc-1885edaf4530 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:17.494981 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14344471-8829-4843-8fcc-1885edaf4530 admin admin] PATH_INFO: `/v3/users/92a073fcff47436f9a623f115b49759e` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:17.517109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14344471-8829-4843-8fcc-1885edaf4530 admin admin] RBAC: Authorizing `identity:delete_user(user_id=92a073fcff47436f9a623f115b49759e)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:17.519776 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14344471-8829-4843-8fcc-1885edaf4530 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:17.615239 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-424e09bc-f768-42e7-84f7-cd0c5a7d78b7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f2a1c22c27f14f7a8d0a7be7265d7b14'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.616403 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-424e09bc-f768-42e7-84f7-cd0c5a7d78b7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f2a1c22c27f14f7a8d0a7be7265d7b14'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.640656 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-424e09bc-f768-42e7-84f7-cd0c5a7d78b7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f2a1c22c27f14f7a8d0a7be7265d7b14'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.643893 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-424e09bc-f768-42e7-84f7-cd0c5a7d78b7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f2a1c22c27f14f7a8d0a7be7265d7b14'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.651651 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-14344471-8829-4843-8fcc-1885edaf4530 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '92a073fcff47436f9a623f115b49759e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.652869 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-14344471-8829-4843-8fcc-1885edaf4530 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '92a073fcff47436f9a623f115b49759e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.667184 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-14344471-8829-4843-8fcc-1885edaf4530 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '92a073fcff47436f9a623f115b49759e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.697546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-424e09bc-f768-42e7-84f7-cd0c5a7d78b7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f2a1c22c27f14f7a8d0a7be7265d7b14'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.733005 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 809/1657] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:17 2022] DELETE /identity/v3/users/f2a1c22c27f14f7a8d0a7be7265d7b14 => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:17.754145 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-14344471-8829-4843-8fcc-1885edaf4530 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '92a073fcff47436f9a623f115b49759e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.772743 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-14344471-8829-4843-8fcc-1885edaf4530 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '92a073fcff47436f9a623f115b49759e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:17.788527 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 849/1658] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:17 2022] DELETE /identity/v3/users/92a073fcff47436f9a623f115b49759e => generated 0 bytes in 568 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:17.945120 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424e09bc-f768-42e7-84f7-cd0c5a7d78b7 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:17.946834 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-424e09bc-f768-42e7-84f7-cd0c5a7d78b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:18.030814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:18.062953 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424e09bc-f768-42e7-84f7-cd0c5a7d78b7 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:18.195218 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4cdbe8-2a26-43ae-8dde-f1d893fa05fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:18.195886 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4cdbe8-2a26-43ae-8dde-f1d893fa05fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:18.196170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4cdbe8-2a26-43ae-8dde-f1d893fa05fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:18.196447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4cdbe8-2a26-43ae-8dde-f1d893fa05fa admin admin] PATH_INFO: `/v3/users/d7ec1d1c4dae476eaa110e2b6dc509e3` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:18.199873 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4cdbe8-2a26-43ae-8dde-f1d893fa05fa admin admin] RBAC: Authorizing `identity:delete_user(user_id=d7ec1d1c4dae476eaa110e2b6dc509e3)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:18.204485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4cdbe8-2a26-43ae-8dde-f1d893fa05fa admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:18.290077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0f4cdbe8-2a26-43ae-8dde-f1d893fa05fa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd7ec1d1c4dae476eaa110e2b6dc509e3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:18.290908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0f4cdbe8-2a26-43ae-8dde-f1d893fa05fa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd7ec1d1c4dae476eaa110e2b6dc509e3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:18.310846 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0f4cdbe8-2a26-43ae-8dde-f1d893fa05fa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd7ec1d1c4dae476eaa110e2b6dc509e3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:18.311700 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0f4cdbe8-2a26-43ae-8dde-f1d893fa05fa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd7ec1d1c4dae476eaa110e2b6dc509e3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:18.337497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14344471-8829-4843-8fcc-1885edaf4530 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:18.340035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-14344471-8829-4843-8fcc-1885edaf4530 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:18.358012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0f4cdbe8-2a26-43ae-8dde-f1d893fa05fa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd7ec1d1c4dae476eaa110e2b6dc509e3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:18.382519 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14344471-8829-4843-8fcc-1885edaf4530 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:18.389915 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 810/1659] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:17 2022] DELETE /identity/v3/users/d7ec1d1c4dae476eaa110e2b6dc509e3 => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:18.395308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fb165e-b984-41a6-8052-a6650844df19 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:18.395738 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fb165e-b984-41a6-8052-a6650844df19 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:18.396379 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fb165e-b984-41a6-8052-a6650844df19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:18.396672 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69171b0-5188-443a-9808-5e6e6f03cb06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:18.397347 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69171b0-5188-443a-9808-5e6e6f03cb06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:18.397625 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69171b0-5188-443a-9808-5e6e6f03cb06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:18.398012 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69171b0-5188-443a-9808-5e6e6f03cb06 admin admin] PATH_INFO: `/v3/projects/e44d58ff3c704162be8230616968b2d6` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:18.401217 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69171b0-5188-443a-9808-5e6e6f03cb06 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e44d58ff3c704162be8230616968b2d6)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:18.403685 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69171b0-5188-443a-9808-5e6e6f03cb06 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:18.457833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-42fb165e-b984-41a6-8052-a6650844df19 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:18.459449 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-42fb165e-b984-41a6-8052-a6650844df19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:18.540646 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b69171b0-5188-443a-9808-5e6e6f03cb06 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e44d58ff3c704162be8230616968b2d6', 'request_id': 'req-b69171b0-5188-443a-9808-5e6e6f03cb06'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:18.559927 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b69171b0-5188-443a-9808-5e6e6f03cb06 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e44d58ff3c704162be8230616968b2d6', 'request_id': 'req-b69171b0-5188-443a-9808-5e6e6f03cb06'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:18.569679 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b69171b0-5188-443a-9808-5e6e6f03cb06 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e44d58ff3c704162be8230616968b2d6', 'request_id': 'req-b69171b0-5188-443a-9808-5e6e6f03cb06'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:18.573970 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 811/1660] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:18.573970 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fb165e-b984-41a6-8052-a6650844df19 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:18.573970 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-42fb165e-b984-41a6-8052-a6650844df19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:18.592757 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fb165e-b984-41a6-8052-a6650844df19 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:18.597808 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:18.598941 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b69171b0-5188-443a-9808-5e6e6f03cb06 admin admin] The token cache is being invalidate because project e44d58ff3c704162be8230616968b2d6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:18.599486 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b69171b0-5188-443a-9808-5e6e6f03cb06 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:18.600647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b69171b0-5188-443a-9808-5e6e6f03cb06 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:18.611002 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 850/1661] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:18 2022] DELETE /identity/v3/projects/e44d58ff3c704162be8230616968b2d6 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:18.611002 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69171b0-5188-443a-9808-5e6e6f03cb06 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:18.611002 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b69171b0-5188-443a-9808-5e6e6f03cb06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:18.648214 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:18.768430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fa41f7-de63-4b38-97bb-c5f10673bd85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:18.769190 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fa41f7-de63-4b38-97bb-c5f10673bd85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:18.769542 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fa41f7-de63-4b38-97bb-c5f10673bd85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:18.769915 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fa41f7-de63-4b38-97bb-c5f10673bd85 admin admin] PATH_INFO: `/v3/projects/75b4f48624b84658bf50d2c95e001e3a` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:18.772731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fa41f7-de63-4b38-97bb-c5f10673bd85 admin admin] RBAC: Authorizing `identity:delete_project(project_id=75b4f48624b84658bf50d2c95e001e3a)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:18.775151 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fa41f7-de63-4b38-97bb-c5f10673bd85 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:18.841520 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69171b0-5188-443a-9808-5e6e6f03cb06 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:18.875437 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-32fa41f7-de63-4b38-97bb-c5f10673bd85 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75b4f48624b84658bf50d2c95e001e3a', 'request_id': 'req-32fa41f7-de63-4b38-97bb-c5f10673bd85'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:18.887623 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-32fa41f7-de63-4b38-97bb-c5f10673bd85 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75b4f48624b84658bf50d2c95e001e3a', 'request_id': 'req-32fa41f7-de63-4b38-97bb-c5f10673bd85'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:18.902098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-32fa41f7-de63-4b38-97bb-c5f10673bd85 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75b4f48624b84658bf50d2c95e001e3a', 'request_id': 'req-32fa41f7-de63-4b38-97bb-c5f10673bd85'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:18.919219 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-32fa41f7-de63-4b38-97bb-c5f10673bd85 admin admin] The token cache is being invalidate because project 75b4f48624b84658bf50d2c95e001e3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:18.919513 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-32fa41f7-de63-4b38-97bb-c5f10673bd85 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:18.920287 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-32fa41f7-de63-4b38-97bb-c5f10673bd85 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:18.926362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 812/1662] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:18 2022] DELETE /identity/v3/projects/75b4f48624b84658bf50d2c95e001e3a => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:18.926362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c73a67c-394d-47dd-8695-0540de270750 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:18.926362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c73a67c-394d-47dd-8695-0540de270750 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:18.926362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c73a67c-394d-47dd-8695-0540de270750 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:18.958853 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4c73a67c-394d-47dd-8695-0540de270750 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:18.961568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4c73a67c-394d-47dd-8695-0540de270750 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:19.000615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e624952-d7fd-4343-bfb9-bc3eb8e41db7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:19.001274 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e624952-d7fd-4343-bfb9-bc3eb8e41db7 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.001535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e624952-d7fd-4343-bfb9-bc3eb8e41db7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.001817 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e624952-d7fd-4343-bfb9-bc3eb8e41db7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.004868 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-3e624952-d7fd-4343-bfb9-bc3eb8e41db7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:19.142642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 813/1663] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:19.144195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a72fab6-78bd-4445-914e-8a6973c72786 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.144616 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a72fab6-78bd-4445-914e-8a6973c72786 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.145139 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a72fab6-78bd-4445-914e-8a6973c72786 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.158429 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e624952-d7fd-4343-bfb9-bc3eb8e41db7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:19.160663 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e624952-d7fd-4343-bfb9-bc3eb8e41db7 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:19.191688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-8a72fab6-78bd-4445-914e-8a6973c72786 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:19.193591 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8a72fab6-78bd-4445-914e-8a6973c72786 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:19.194165 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 851/1664] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 590 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:19.195617 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206a9178-41d5-49a2-9900-5c0a473ab719 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.195902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206a9178-41d5-49a2-9900-5c0a473ab719 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.196183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206a9178-41d5-49a2-9900-5c0a473ab719 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.221068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 814/1665] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:19.232469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8e04c6-7a7c-4c79-9f6d-1687bb9904db None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.232815 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8e04c6-7a7c-4c79-9f6d-1687bb9904db None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.235177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8e04c6-7a7c-4c79-9f6d-1687bb9904db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.251060 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-206a9178-41d5-49a2-9900-5c0a473ab719 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:19.252332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-206a9178-41d5-49a2-9900-5c0a473ab719 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:19.290222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 852/1666] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:19.296401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206a9178-41d5-49a2-9900-5c0a473ab719 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:19.308112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-0a8e04c6-7a7c-4c79-9f6d-1687bb9904db None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:19.309119 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0a8e04c6-7a7c-4c79-9f6d-1687bb9904db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:19.325317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206a9178-41d5-49a2-9900-5c0a473ab719 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:19.337203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 815/1667] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:19.340890 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0d9aa5-4115-4e9c-a8a9-64499a0f6d1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:19.342138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8e04c6-7a7c-4c79-9f6d-1687bb9904db None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:19.345201 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0d9aa5-4115-4e9c-a8a9-64499a0f6d1d admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.345484 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0d9aa5-4115-4e9c-a8a9-64499a0f6d1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.345755 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0d9aa5-4115-4e9c-a8a9-64499a0f6d1d admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.346444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0d9aa5-4115-4e9c-a8a9-64499a0f6d1d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:58:19.346775 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0d9aa5-4115-4e9c-a8a9-64499a0f6d1d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:19.350420 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0d9aa5-4115-4e9c-a8a9-64499a0f6d1d admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:19.367824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8e04c6-7a7c-4c79-9f6d-1687bb9904db None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:19.382929 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ca04f3-63a3-4c90-8199-e81cfcb151fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:19.383977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ca04f3-63a3-4c90-8199-e81cfcb151fa admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.384562 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ca04f3-63a3-4c90-8199-e81cfcb151fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.384724 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ca04f3-63a3-4c90-8199-e81cfcb151fa admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.385686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ca04f3-63a3-4c90-8199-e81cfcb151fa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:58:19.386791 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ca04f3-63a3-4c90-8199-e81cfcb151fa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:19.388944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ca04f3-63a3-4c90-8199-e81cfcb151fa admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:19.394579 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 853/1668] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:58:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:58:19.395728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0d9aa5-4115-4e9c-a8a9-64499a0f6d1d admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:19.417986 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 816/1669] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:58:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:58:19.419404 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ca04f3-63a3-4c90-8199-e81cfcb151fa admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:19.420057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0d9aa5-4115-4e9c-a8a9-64499a0f6d1d admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:19.432428 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fec1e8-2c3e-42ed-8d3b-ee49d98fbda6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:19.432859 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ca04f3-63a3-4c90-8199-e81cfcb151fa admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:19.433666 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fec1e8-2c3e-42ed-8d3b-ee49d98fbda6 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.434947 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fec1e8-2c3e-42ed-8d3b-ee49d98fbda6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.434947 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fec1e8-2c3e-42ed-8d3b-ee49d98fbda6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.435158 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fec1e8-2c3e-42ed-8d3b-ee49d98fbda6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:19.436949 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fec1e8-2c3e-42ed-8d3b-ee49d98fbda6 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:19.440349 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268030d1-65d7-4a38-855b-1fef994404ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:19.441163 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268030d1-65d7-4a38-855b-1fef994404ee admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.441440 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268030d1-65d7-4a38-855b-1fef994404ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.441739 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268030d1-65d7-4a38-855b-1fef994404ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.442454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268030d1-65d7-4a38-855b-1fef994404ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:19.444528 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268030d1-65d7-4a38-855b-1fef994404ee admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:19.457951 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:19.492426 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 854/1670] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:19 2022] POST /identity/v3/projects => generated 373 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:58:19.497792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fec1e8-2c3e-42ed-8d3b-ee49d98fbda6 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:19.498120 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 817/1671] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:19 2022] POST /identity/v3/projects => generated 407 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:19.502570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268030d1-65d7-4a38-855b-1fef994404ee admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:19.582111 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:19.587215 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268030d1-65d7-4a38-855b-1fef994404ee admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:19.594050 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27eab27b-3663-4ba5-ab90-737dbd135679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:19.594904 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eab27b-3663-4ba5-ab90-737dbd135679 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.595196 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eab27b-3663-4ba5-ab90-737dbd135679 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.595477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eab27b-3663-4ba5-ab90-737dbd135679 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.596510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27eab27b-3663-4ba5-ab90-737dbd135679 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:19.599301 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27eab27b-3663-4ba5-ab90-737dbd135679 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:19.619569 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fec1e8-2c3e-42ed-8d3b-ee49d98fbda6 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:19.626305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5da6269-7c6f-42a0-96bf-01de69839d6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:19.627163 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5da6269-7c6f-42a0-96bf-01de69839d6b admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.627822 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 818/1672] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:19 2022] POST /identity/v3/users => generated 304 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:58:19.630572 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5da6269-7c6f-42a0-96bf-01de69839d6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.631221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5da6269-7c6f-42a0-96bf-01de69839d6b admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.631998 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5da6269-7c6f-42a0-96bf-01de69839d6b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:19.633494 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27eab27b-3663-4ba5-ab90-737dbd135679 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:19.638233 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5da6269-7c6f-42a0-96bf-01de69839d6b admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:19.656336 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27eab27b-3663-4ba5-ab90-737dbd135679 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:19.669573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 855/1673] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:19 2022] POST /identity/v3/users => generated 287 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:19.673393 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5da6269-7c6f-42a0-96bf-01de69839d6b admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:19.674467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd65e076-2aac-4175-8735-d83ba32e9bd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:19.675123 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd65e076-2aac-4175-8735-d83ba32e9bd3 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.675392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd65e076-2aac-4175-8735-d83ba32e9bd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.675661 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd65e076-2aac-4175-8735-d83ba32e9bd3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.676310 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd65e076-2aac-4175-8735-d83ba32e9bd3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:19.677909 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd65e076-2aac-4175-8735-d83ba32e9bd3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:19.689380 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5da6269-7c6f-42a0-96bf-01de69839d6b admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:19.699191 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 819/1674] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:19 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:19.703416 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd65e076-2aac-4175-8735-d83ba32e9bd3 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:19.709108 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30f94dc-89f4-4acd-bdda-c74fd08364bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:19.709749 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30f94dc-89f4-4acd-bdda-c74fd08364bc admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.710026 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30f94dc-89f4-4acd-bdda-c74fd08364bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.710303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30f94dc-89f4-4acd-bdda-c74fd08364bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.710997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30f94dc-89f4-4acd-bdda-c74fd08364bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:19.712587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30f94dc-89f4-4acd-bdda-c74fd08364bc admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:19.726352 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd65e076-2aac-4175-8735-d83ba32e9bd3 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:19.735569 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34920bf9-c2a5-4de4-9d9b-e8739dff17bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:19.735812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34920bf9-c2a5-4de4-9d9b-e8739dff17bd admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.736035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34920bf9-c2a5-4de4-9d9b-e8739dff17bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.736264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34920bf9-c2a5-4de4-9d9b-e8739dff17bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.737221 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34920bf9-c2a5-4de4-9d9b-e8739dff17bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:19.739391 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34920bf9-c2a5-4de4-9d9b-e8739dff17bd admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:19.743609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 856/1675] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:19 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 00:58:19.744580 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30f94dc-89f4-4acd-bdda-c74fd08364bc admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:19.759370 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 820/1676] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:19 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:19.763590 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34920bf9-c2a5-4de4-9d9b-e8739dff17bd admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:19.776949 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30f94dc-89f4-4acd-bdda-c74fd08364bc admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:19.787612 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34920bf9-c2a5-4de4-9d9b-e8739dff17bd admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:19.788986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0a558b-7382-42b4-aff2-1ce77f1098b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:19.790287 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0a558b-7382-42b4-aff2-1ce77f1098b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.790576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0a558b-7382-42b4-aff2-1ce77f1098b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.790852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0a558b-7382-42b4-aff2-1ce77f1098b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.794618 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0a558b-7382-42b4-aff2-1ce77f1098b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:19.797872 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0a558b-7382-42b4-aff2-1ce77f1098b6 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:19.806174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d76b7a8-58be-4980-9a8c-88ece2e21a84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:19.806174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d76b7a8-58be-4980-9a8c-88ece2e21a84 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.806174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d76b7a8-58be-4980-9a8c-88ece2e21a84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.806174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d76b7a8-58be-4980-9a8c-88ece2e21a84 admin admin] PATH_INFO: `/v3/projects/1fbf3a4d437f48a0bab6f7d5ed18d307/users/393d97606252471f88212c8555108afe/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.832516 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 857/1677] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:19 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:19.834465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0a558b-7382-42b4-aff2-1ce77f1098b6 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:19.858422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d76b7a8-58be-4980-9a8c-88ece2e21a84 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1fbf3a4d437f48a0bab6f7d5ed18d307, user_id=393d97606252471f88212c8555108afe, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:19.861320 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d76b7a8-58be-4980-9a8c-88ece2e21a84 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:19.880468 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0a558b-7382-42b4-aff2-1ce77f1098b6 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:19.888244 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b17800-0520-42bf-b067-b394e11311ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:19.888993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b17800-0520-42bf-b067-b394e11311ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.889306 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b17800-0520-42bf-b067-b394e11311ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.889584 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b17800-0520-42bf-b067-b394e11311ed admin admin] PATH_INFO: `/v3/projects/39d1c20705604f8794fb441271e5afd3/users/86fccabd87a84389b219b00cdcc857e2/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.901173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 821/1678] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:19 2022] PUT /identity/v3/projects/1fbf3a4d437f48a0bab6f7d5ed18d307/users/393d97606252471f88212c8555108afe/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:19.913213 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0259e9e-085c-4e1d-a88c-602a98968fd8 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.913568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0259e9e-085c-4e1d-a88c-602a98968fd8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.913971 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0259e9e-085c-4e1d-a88c-602a98968fd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.921821 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b17800-0520-42bf-b067-b394e11311ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=39d1c20705604f8794fb441271e5afd3, user_id=86fccabd87a84389b219b00cdcc857e2, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:19.925475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b17800-0520-42bf-b067-b394e11311ed admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:19.980687 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 858/1679] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:19 2022] PUT /identity/v3/projects/39d1c20705604f8794fb441271e5afd3/users/86fccabd87a84389b219b00cdcc857e2/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:19.985553 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f5707a-2eea-41b4-9eb5-e5c9b66af676 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:19.985845 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f5707a-2eea-41b4-9eb5-e5c9b66af676 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:19.986141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f5707a-2eea-41b4-9eb5-e5c9b66af676 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:19.990572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f0259e9e-085c-4e1d-a88c-602a98968fd8 None None] MFA Rules not processed for user `393d97606252471f88212c8555108afe`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:19.992667 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f0259e9e-085c-4e1d-a88c-602a98968fd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:20.028154 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-d8f5707a-2eea-41b4-9eb5-e5c9b66af676 None None] MFA Rules not processed for user `86fccabd87a84389b219b00cdcc857e2`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:20.029480 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d8f5707a-2eea-41b4-9eb5-e5c9b66af676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:20.096282 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 822/1680] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:19 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:58:20.106390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0259e9e-085c-4e1d-a88c-602a98968fd8 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:20.158501 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:20.167344 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 859/1681] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:19 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:20.167344 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f5707a-2eea-41b4-9eb5-e5c9b66af676 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:20.177209 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d8f5707a-2eea-41b4-9eb5-e5c9b66af676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:20.209432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0259e9e-085c-4e1d-a88c-602a98968fd8 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:20.215301 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68893148-727b-44f9-90d9-33a621126d97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:20.215937 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68893148-727b-44f9-90d9-33a621126d97 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:20.216273 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68893148-727b-44f9-90d9-33a621126d97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:20.216543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68893148-727b-44f9-90d9-33a621126d97 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:20.217136 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68893148-727b-44f9-90d9-33a621126d97 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:20.219109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68893148-727b-44f9-90d9-33a621126d97 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:20.256556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 823/1682] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:20 2022] POST /identity/v3/projects => generated 407 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:20.257492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68893148-727b-44f9-90d9-33a621126d97 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:20.259266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-68893148-727b-44f9-90d9-33a621126d97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:20.259871 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f5707a-2eea-41b4-9eb5-e5c9b66af676 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:20.274190 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:20.329782 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:20.330450 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:20.330724 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:20.331034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c admin admin] PATH_INFO: `/v3/projects/69722a0b3c1440de98c6e392d1b120c7` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:20.332537 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c admin admin] RBAC: Authorizing `identity:delete_project(project_id=69722a0b3c1440de98c6e392d1b120c7)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:20.334153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68893148-727b-44f9-90d9-33a621126d97 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:20.334487 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:20.337919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff771b1e-af25-455d-bce8-2b7d653287d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:20.338561 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff771b1e-af25-455d-bce8-2b7d653287d1 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:20.338877 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff771b1e-af25-455d-bce8-2b7d653287d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:20.339842 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff771b1e-af25-455d-bce8-2b7d653287d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:20.421922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '69722a0b3c1440de98c6e392d1b120c7', 'request_id': 'req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:20.436523 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff771b1e-af25-455d-bce8-2b7d653287d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:20.438587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff771b1e-af25-455d-bce8-2b7d653287d1 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:20.439400 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '69722a0b3c1440de98c6e392d1b120c7', 'request_id': 'req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:20.452683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 824/1683] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:20 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:20.453357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '69722a0b3c1440de98c6e392d1b120c7', 'request_id': 'req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:20.454253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff771b1e-af25-455d-bce8-2b7d653287d1 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:20.467351 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff771b1e-af25-455d-bce8-2b7d653287d1 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:20.492415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c admin admin] The token cache is being invalidate because project 69722a0b3c1440de98c6e392d1b120c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:20.492779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:20.494759 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:20.499010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 860/1684] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:20 2022] DELETE /identity/v3/projects/69722a0b3c1440de98c6e392d1b120c7 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:20.503102 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:20.508598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:20.557357 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6934e1a-b08f-43aa-ba4f-74826466cd7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:20.558465 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6934e1a-b08f-43aa-ba4f-74826466cd7d admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:20.558746 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6934e1a-b08f-43aa-ba4f-74826466cd7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:20.559267 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6934e1a-b08f-43aa-ba4f-74826466cd7d admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:20.561556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6934e1a-b08f-43aa-ba4f-74826466cd7d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:20.569587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6934e1a-b08f-43aa-ba4f-74826466cd7d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:20.597882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:20.621445 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 825/1685] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:20 2022] POST /identity/v3/users => generated 310 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:20.627835 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6934e1a-b08f-43aa-ba4f-74826466cd7d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:20.633905 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b6934e1a-b08f-43aa-ba4f-74826466cd7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:20.660805 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6934e1a-b08f-43aa-ba4f-74826466cd7d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:20.669851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c3326f-667d-43e2-a614-dd0403dea0a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:20.670898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c3326f-667d-43e2-a614-dd0403dea0a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:20.670898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c3326f-667d-43e2-a614-dd0403dea0a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:20.671094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c3326f-667d-43e2-a614-dd0403dea0a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:20.671586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c3326f-667d-43e2-a614-dd0403dea0a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:20.673438 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c3326f-667d-43e2-a614-dd0403dea0a0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:20.681102 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a877da-3f0d-4c4d-ae1e-4de5fa43447c admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:20.690798 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc4f8ff-aacf-43b8-bfe4-593554247dfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:20.691583 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc4f8ff-aacf-43b8-bfe4-593554247dfc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:20.691974 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc4f8ff-aacf-43b8-bfe4-593554247dfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:20.693180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc4f8ff-aacf-43b8-bfe4-593554247dfc admin admin] PATH_INFO: `/v3/users/9ee9c70f18b74717b51f4230940e3aad` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:20.696504 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc4f8ff-aacf-43b8-bfe4-593554247dfc admin admin] RBAC: Authorizing `identity:delete_user(user_id=9ee9c70f18b74717b51f4230940e3aad)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:20.699150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc4f8ff-aacf-43b8-bfe4-593554247dfc admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:20.709332 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 826/1686] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:20 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:20.712165 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c3326f-667d-43e2-a614-dd0403dea0a0 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:20.736662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c3326f-667d-43e2-a614-dd0403dea0a0 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:20.747244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53eef82d-30ae-42ef-a0e8-50dd624c5d97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:20.747886 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53eef82d-30ae-42ef-a0e8-50dd624c5d97 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:20.748157 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53eef82d-30ae-42ef-a0e8-50dd624c5d97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:20.748444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53eef82d-30ae-42ef-a0e8-50dd624c5d97 admin admin] PATH_INFO: `/v3/projects/9eca102f1d9749a3a2723f07db30f289/users/3bce99e644e4411e98b6b51b4f8a1fea/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:20.769159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53eef82d-30ae-42ef-a0e8-50dd624c5d97 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9eca102f1d9749a3a2723f07db30f289, user_id=3bce99e644e4411e98b6b51b4f8a1fea, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:20.772903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53eef82d-30ae-42ef-a0e8-50dd624c5d97 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:20.790509 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-acc4f8ff-aacf-43b8-bfe4-593554247dfc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9ee9c70f18b74717b51f4230940e3aad'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:20.791706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-acc4f8ff-aacf-43b8-bfe4-593554247dfc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9ee9c70f18b74717b51f4230940e3aad'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:20.795148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 827/1687] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:20 2022] PUT /identity/v3/projects/9eca102f1d9749a3a2723f07db30f289/users/3bce99e644e4411e98b6b51b4f8a1fea/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:20.797205 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:20.801153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53eef82d-30ae-42ef-a0e8-50dd624c5d97 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:20.806288 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-acc4f8ff-aacf-43b8-bfe4-593554247dfc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9ee9c70f18b74717b51f4230940e3aad'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:20.836537 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-acc4f8ff-aacf-43b8-bfe4-593554247dfc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9ee9c70f18b74717b51f4230940e3aad'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:20.849202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-acc4f8ff-aacf-43b8-bfe4-593554247dfc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9ee9c70f18b74717b51f4230940e3aad'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:20.868762 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 861/1688] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:20 2022] DELETE /identity/v3/users/9ee9c70f18b74717b51f4230940e3aad => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:20.919481 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53eef82d-30ae-42ef-a0e8-50dd624c5d97 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:21.038082 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fa531e-eae1-4139-b2e4-378c42e2e2dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:21.044291 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fa531e-eae1-4139-b2e4-378c42e2e2dc admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:21.044754 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fa531e-eae1-4139-b2e4-378c42e2e2dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:21.045042 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fa531e-eae1-4139-b2e4-378c42e2e2dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:21.045716 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fa531e-eae1-4139-b2e4-378c42e2e2dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:21.047371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fa531e-eae1-4139-b2e4-378c42e2e2dc admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:21.059679 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 828/1689] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:20 2022] GET /identity/v3/roles => generated 1420 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:21.064335 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc4f8ff-aacf-43b8-bfe4-593554247dfc admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:21.066482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-acc4f8ff-aacf-43b8-bfe4-593554247dfc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:21.081504 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc4f8ff-aacf-43b8-bfe4-593554247dfc admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:21.087850 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be626efd-7948-41dc-b5d2-f5ace1d5e83c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:21.088540 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be626efd-7948-41dc-b5d2-f5ace1d5e83c admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:21.088821 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be626efd-7948-41dc-b5d2-f5ace1d5e83c admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:21.089712 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be626efd-7948-41dc-b5d2-f5ace1d5e83c admin admin] PATH_INFO: `/v3/projects/9eca102f1d9749a3a2723f07db30f289/users/3bce99e644e4411e98b6b51b4f8a1fea/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:21.091867 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be626efd-7948-41dc-b5d2-f5ace1d5e83c admin admin] RBAC: Authorizing `identity:create_grant(project_id=9eca102f1d9749a3a2723f07db30f289, user_id=3bce99e644e4411e98b6b51b4f8a1fea, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:21.094631 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be626efd-7948-41dc-b5d2-f5ace1d5e83c admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:21.111137 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 862/1690] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:21 2022] PUT /identity/v3/projects/9eca102f1d9749a3a2723f07db30f289/users/3bce99e644e4411e98b6b51b4f8a1fea/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:21.126710 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4263e580-987b-45c2-aed9-197b66a31086 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:21.127036 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4263e580-987b-45c2-aed9-197b66a31086 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:21.127313 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4263e580-987b-45c2-aed9-197b66a31086 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:21.162164 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4263e580-987b-45c2-aed9-197b66a31086 None None] MFA Rules not processed for user `3bce99e644e4411e98b6b51b4f8a1fea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:21.163371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4263e580-987b-45c2-aed9-197b66a31086 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:21.279785 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:21.317443 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be626efd-7948-41dc-b5d2-f5ace1d5e83c admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:21.320015 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-be626efd-7948-41dc-b5d2-f5ace1d5e83c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:21.353767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 829/1691] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:21 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:21.366931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4263e580-987b-45c2-aed9-197b66a31086 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:21.368804 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4263e580-987b-45c2-aed9-197b66a31086 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:21.457333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4263e580-987b-45c2-aed9-197b66a31086 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:21.462467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d43548-8058-4617-8b76-fcf62fc6a0b4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:21.465701 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d43548-8058-4617-8b76-fcf62fc6a0b4 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:21.466376 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d43548-8058-4617-8b76-fcf62fc6a0b4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:21.466667 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d43548-8058-4617-8b76-fcf62fc6a0b4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:21.469588 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be626efd-7948-41dc-b5d2-f5ace1d5e83c admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:21.477486 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c022ee5-c898-43bd-8401-4e2b3a05c53d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:21.478776 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c022ee5-c898-43bd-8401-4e2b3a05c53d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:21.479153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c022ee5-c898-43bd-8401-4e2b3a05c53d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:21.479431 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c022ee5-c898-43bd-8401-4e2b3a05c53d admin admin] PATH_INFO: `/v3/projects/039a813f1de5494cbd44cc85ede9fbf8` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:21.481383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c022ee5-c898-43bd-8401-4e2b3a05c53d admin admin] RBAC: Authorizing `identity:delete_project(project_id=039a813f1de5494cbd44cc85ede9fbf8)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:21.484071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c022ee5-c898-43bd-8401-4e2b3a05c53d admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:21.484686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d43548-8058-4617-8b76-fcf62fc6a0b4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:21.486499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d43548-8058-4617-8b76-fcf62fc6a0b4 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:21.503844 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 830/1692] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:21 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:21.544732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-8c022ee5-c898-43bd-8401-4e2b3a05c53d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '039a813f1de5494cbd44cc85ede9fbf8', 'request_id': 'req-8c022ee5-c898-43bd-8401-4e2b3a05c53d'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:21.571442 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-8c022ee5-c898-43bd-8401-4e2b3a05c53d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '039a813f1de5494cbd44cc85ede9fbf8', 'request_id': 'req-8c022ee5-c898-43bd-8401-4e2b3a05c53d'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:21.581910 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-8c022ee5-c898-43bd-8401-4e2b3a05c53d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '039a813f1de5494cbd44cc85ede9fbf8', 'request_id': 'req-8c022ee5-c898-43bd-8401-4e2b3a05c53d'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:21.597601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-8c022ee5-c898-43bd-8401-4e2b3a05c53d admin admin] The token cache is being invalidate because project 039a813f1de5494cbd44cc85ede9fbf8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:21.598069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-8c022ee5-c898-43bd-8401-4e2b3a05c53d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:21.601205 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-8c022ee5-c898-43bd-8401-4e2b3a05c53d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:21.606909 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 863/1693] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:21 2022] DELETE /identity/v3/projects/039a813f1de5494cbd44cc85ede9fbf8 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:21.608723 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d43548-8058-4617-8b76-fcf62fc6a0b4 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:21.622190 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d43548-8058-4617-8b76-fcf62fc6a0b4 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:21.706112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740016d3-6e0c-4553-813f-cd3577b816b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:21.707068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740016d3-6e0c-4553-813f-cd3577b816b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:21.707825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740016d3-6e0c-4553-813f-cd3577b816b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:21.708285 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740016d3-6e0c-4553-813f-cd3577b816b6 admin admin] PATH_INFO: `/v3/users/d076aa56efb748f0becaea215cf915b7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:21.709779 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740016d3-6e0c-4553-813f-cd3577b816b6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d076aa56efb748f0becaea215cf915b7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:21.712690 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740016d3-6e0c-4553-813f-cd3577b816b6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:21.728795 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c022ee5-c898-43bd-8401-4e2b3a05c53d admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:21.730775 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8c022ee5-c898-43bd-8401-4e2b3a05c53d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:21.781339 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-740016d3-6e0c-4553-813f-cd3577b816b6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd076aa56efb748f0becaea215cf915b7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:21.782379 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-740016d3-6e0c-4553-813f-cd3577b816b6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd076aa56efb748f0becaea215cf915b7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:21.792902 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-740016d3-6e0c-4553-813f-cd3577b816b6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd076aa56efb748f0becaea215cf915b7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:21.793748 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-740016d3-6e0c-4553-813f-cd3577b816b6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd076aa56efb748f0becaea215cf915b7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:21.812509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-740016d3-6e0c-4553-813f-cd3577b816b6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd076aa56efb748f0becaea215cf915b7'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:21.821148 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:DELETE=10,keystone:SELECT=88,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:21.829907 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 831/1694] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:21 2022] DELETE /identity/v3/users/d076aa56efb748f0becaea215cf915b7 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:21.844138 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c022ee5-c898-43bd-8401-4e2b3a05c53d admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:21.949935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311f364e-14fc-42fc-8db4-93806d5ff637 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:21.953354 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311f364e-14fc-42fc-8db4-93806d5ff637 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:21.953654 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311f364e-14fc-42fc-8db4-93806d5ff637 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:21.954038 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311f364e-14fc-42fc-8db4-93806d5ff637 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:21.960764 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-311f364e-14fc-42fc-8db4-93806d5ff637 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:22.081358 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311f364e-14fc-42fc-8db4-93806d5ff637 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:22.083257 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311f364e-14fc-42fc-8db4-93806d5ff637 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:22.095792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 864/1695] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:21 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:22.241102 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740016d3-6e0c-4553-813f-cd3577b816b6 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:22.242702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-740016d3-6e0c-4553-813f-cd3577b816b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:22.259617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:22.441313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8f2224-4803-45be-9379-9122e4d41d20 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:22.441608 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8f2224-4803-45be-9379-9122e4d41d20 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:22.441894 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8f2224-4803-45be-9379-9122e4d41d20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:22.478834 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740016d3-6e0c-4553-813f-cd3577b816b6 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:22.482216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-ff8f2224-4803-45be-9379-9122e4d41d20 None None] MFA Rules not processed for user `86fccabd87a84389b219b00cdcc857e2`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:22.486950 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-ff8f2224-4803-45be-9379-9122e4d41d20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:22.506497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b2e7a1-494a-4558-81c8-08b9c3b17b5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:22.509579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b2e7a1-494a-4558-81c8-08b9c3b17b5b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:22.510058 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b2e7a1-494a-4558-81c8-08b9c3b17b5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:22.510530 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b2e7a1-494a-4558-81c8-08b9c3b17b5b admin admin] PATH_INFO: `/v3/projects/978a7bd015b148ff81040c266cdcf551` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:22.516212 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b2e7a1-494a-4558-81c8-08b9c3b17b5b admin admin] RBAC: Authorizing `identity:delete_project(project_id=978a7bd015b148ff81040c266cdcf551)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:22.523094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b2e7a1-494a-4558-81c8-08b9c3b17b5b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:22.622547 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 865/1696] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:22 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:58:22.640824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-09b2e7a1-494a-4558-81c8-08b9c3b17b5b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '978a7bd015b148ff81040c266cdcf551', 'request_id': 'req-09b2e7a1-494a-4558-81c8-08b9c3b17b5b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:22.654062 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-09b2e7a1-494a-4558-81c8-08b9c3b17b5b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '978a7bd015b148ff81040c266cdcf551', 'request_id': 'req-09b2e7a1-494a-4558-81c8-08b9c3b17b5b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:22.666616 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-09b2e7a1-494a-4558-81c8-08b9c3b17b5b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '978a7bd015b148ff81040c266cdcf551', 'request_id': 'req-09b2e7a1-494a-4558-81c8-08b9c3b17b5b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:22.688262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-09b2e7a1-494a-4558-81c8-08b9c3b17b5b admin admin] The token cache is being invalidate because project 978a7bd015b148ff81040c266cdcf551 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:22.688627 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-09b2e7a1-494a-4558-81c8-08b9c3b17b5b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:22.689352 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-09b2e7a1-494a-4558-81c8-08b9c3b17b5b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:22.691499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 832/1697] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:22 2022] DELETE /identity/v3/projects/978a7bd015b148ff81040c266cdcf551 => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:58:22.700779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc9c55e-6b46-4dd6-ab82-b8f2c59524f9 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:22.701707 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc9c55e-6b46-4dd6-ab82-b8f2c59524f9 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:22.703509 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc9c55e-6b46-4dd6-ab82-b8f2c59524f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:22.735823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-9bc9c55e-6b46-4dd6-ab82-b8f2c59524f9 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:22.739710 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-9bc9c55e-6b46-4dd6-ab82-b8f2c59524f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:22.840710 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:22.859213 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 866/1698] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:58:22.867858 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1962734e-f999-4cfc-ba7f-8678813b45dd None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:22.868150 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1962734e-f999-4cfc-ba7f-8678813b45dd None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:22.868478 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1962734e-f999-4cfc-ba7f-8678813b45dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:22.896653 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-1962734e-f999-4cfc-ba7f-8678813b45dd None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:22.897782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1962734e-f999-4cfc-ba7f-8678813b45dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:22.909109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 833/1699] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:58:22.913847 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc9c55e-6b46-4dd6-ab82-b8f2c59524f9 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:22.926720 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc9c55e-6b46-4dd6-ab82-b8f2c59524f9 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:22.932504 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2115c4d1-d9e4-4e7a-95fe-1294cb3d675f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:22.934081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2115c4d1-d9e4-4e7a-95fe-1294cb3d675f admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:22.934439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2115c4d1-d9e4-4e7a-95fe-1294cb3d675f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:22.934808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2115c4d1-d9e4-4e7a-95fe-1294cb3d675f admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:22.935640 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2115c4d1-d9e4-4e7a-95fe-1294cb3d675f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:58:22.936017 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2115c4d1-d9e4-4e7a-95fe-1294cb3d675f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:22.937747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2115c4d1-d9e4-4e7a-95fe-1294cb3d675f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:22.951687 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 867/1700] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:58:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:58:22.956943 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1962734e-f999-4cfc-ba7f-8678813b45dd None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:22.967152 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1962734e-f999-4cfc-ba7f-8678813b45dd None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:22.974243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abce4a69-b09d-4dc0-ac44-44cc62719678 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:22.974914 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abce4a69-b09d-4dc0-ac44-44cc62719678 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:22.975202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abce4a69-b09d-4dc0-ac44-44cc62719678 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:22.976076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abce4a69-b09d-4dc0-ac44-44cc62719678 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:22.976199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abce4a69-b09d-4dc0-ac44-44cc62719678 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:22.978161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abce4a69-b09d-4dc0-ac44-44cc62719678 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:23.029863 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 834/1701] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:22 2022] POST /identity/v3/projects => generated 377 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:23.034469 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2115c4d1-d9e4-4e7a-95fe-1294cb3d675f admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:23.047010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2115c4d1-d9e4-4e7a-95fe-1294cb3d675f admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:23.148316 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3dbde3-d982-47ea-952c-5aa92a3d2772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:23.149291 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3dbde3-d982-47ea-952c-5aa92a3d2772 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:23.149578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3dbde3-d982-47ea-952c-5aa92a3d2772 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:23.149859 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3dbde3-d982-47ea-952c-5aa92a3d2772 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:23.150526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3dbde3-d982-47ea-952c-5aa92a3d2772 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:23.154299 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3dbde3-d982-47ea-952c-5aa92a3d2772 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:23.186483 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 868/1702] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:23 2022] POST /identity/v3/users => generated 289 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:23.190653 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abce4a69-b09d-4dc0-ac44-44cc62719678 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:23.204662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abce4a69-b09d-4dc0-ac44-44cc62719678 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:23.213379 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02e910a-0ac6-47b5-84ba-e3594b036c4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:23.215313 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02e910a-0ac6-47b5-84ba-e3594b036c4e admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:23.215773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02e910a-0ac6-47b5-84ba-e3594b036c4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:23.216198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02e910a-0ac6-47b5-84ba-e3594b036c4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:23.217203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02e910a-0ac6-47b5-84ba-e3594b036c4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:23.219777 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02e910a-0ac6-47b5-84ba-e3594b036c4e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:23.232878 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 835/1703] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:23 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:23.237080 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3dbde3-d982-47ea-952c-5aa92a3d2772 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:23.248849 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3dbde3-d982-47ea-952c-5aa92a3d2772 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:23.253964 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe39eec-38e0-4716-8f70-91591df97d9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:23.255841 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe39eec-38e0-4716-8f70-91591df97d9f admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:23.255841 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe39eec-38e0-4716-8f70-91591df97d9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:23.255841 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe39eec-38e0-4716-8f70-91591df97d9f admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:23.256764 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe39eec-38e0-4716-8f70-91591df97d9f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:23.258525 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe39eec-38e0-4716-8f70-91591df97d9f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:23.271998 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 869/1704] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:23 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:23.276016 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02e910a-0ac6-47b5-84ba-e3594b036c4e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:23.288021 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02e910a-0ac6-47b5-84ba-e3594b036c4e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:23.295630 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560e7fab-1052-4456-aa1b-f17aed14695d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:23.296798 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560e7fab-1052-4456-aa1b-f17aed14695d admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:23.297081 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560e7fab-1052-4456-aa1b-f17aed14695d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:23.297571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560e7fab-1052-4456-aa1b-f17aed14695d admin admin] PATH_INFO: `/v3/projects/d49d0817c6b74a108b6bb1499b6fd482/users/f841074dc67143acafcb151baf963113/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:23.314490 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560e7fab-1052-4456-aa1b-f17aed14695d admin admin] RBAC: Authorizing `identity:create_grant(project_id=d49d0817c6b74a108b6bb1499b6fd482, user_id=f841074dc67143acafcb151baf963113, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:23.317266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560e7fab-1052-4456-aa1b-f17aed14695d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:23.334930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 836/1705] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:23 2022] PUT /identity/v3/projects/d49d0817c6b74a108b6bb1499b6fd482/users/f841074dc67143acafcb151baf963113/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:23.347859 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941b0c1f-0dc4-4684-b431-9f3dad2ced76 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:23.348154 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941b0c1f-0dc4-4684-b431-9f3dad2ced76 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:23.348430 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941b0c1f-0dc4-4684-b431-9f3dad2ced76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:23.382554 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-941b0c1f-0dc4-4684-b431-9f3dad2ced76 None None] MFA Rules not processed for user `f841074dc67143acafcb151baf963113`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:23.385209 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-941b0c1f-0dc4-4684-b431-9f3dad2ced76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:23.487116 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 870/1706] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:23 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:23.500272 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560e7fab-1052-4456-aa1b-f17aed14695d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:23.504466 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-560e7fab-1052-4456-aa1b-f17aed14695d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:23.554582 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:23.624427 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560e7fab-1052-4456-aa1b-f17aed14695d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:23.629198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa1f95e-ea1b-4df5-8848-74625592e530 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '78473c29cef449b89bc9185de67f1bf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:23.630081 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa1f95e-ea1b-4df5-8848-74625592e530 service glance] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:23.630548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa1f95e-ea1b-4df5-8848-74625592e530 service glance] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:23.635248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa1f95e-ea1b-4df5-8848-74625592e530 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:23.646197 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa1f95e-ea1b-4df5-8848-74625592e530 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:23.647794 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa1f95e-ea1b-4df5-8848-74625592e530 service glance] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:23.661401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 837/1707] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:23 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:31.183277 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941b0c1f-0dc4-4684-b431-9f3dad2ced76 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:31.186669 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-941b0c1f-0dc4-4684-b431-9f3dad2ced76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:31.325730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:31.360554 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941b0c1f-0dc4-4684-b431-9f3dad2ced76 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:31.368478 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1e0eae-c0ad-45fe-8f65-20f85228ab8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:31.369318 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1e0eae-c0ad-45fe-8f65-20f85228ab8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:31.369794 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1e0eae-c0ad-45fe-8f65-20f85228ab8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:31.370274 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1e0eae-c0ad-45fe-8f65-20f85228ab8f admin admin] PATH_INFO: `/v3/users/393d97606252471f88212c8555108afe` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:31.373050 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1e0eae-c0ad-45fe-8f65-20f85228ab8f admin admin] RBAC: Authorizing `identity:delete_user(user_id=393d97606252471f88212c8555108afe)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:31.376587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1e0eae-c0ad-45fe-8f65-20f85228ab8f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:31.499141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-db1e0eae-c0ad-45fe-8f65-20f85228ab8f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '393d97606252471f88212c8555108afe'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:31.499925 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-db1e0eae-c0ad-45fe-8f65-20f85228ab8f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '393d97606252471f88212c8555108afe'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:31.520905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-db1e0eae-c0ad-45fe-8f65-20f85228ab8f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '393d97606252471f88212c8555108afe'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:31.539989 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-db1e0eae-c0ad-45fe-8f65-20f85228ab8f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '393d97606252471f88212c8555108afe'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:31.574916 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-db1e0eae-c0ad-45fe-8f65-20f85228ab8f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '393d97606252471f88212c8555108afe'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:31.605941 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 871/1708] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:31 2022] DELETE /identity/v3/users/393d97606252471f88212c8555108afe => generated 0 bytes in 424 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:31.612181 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2c92c9-a703-41d0-90c6-9cdda5555cf7 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:31.613739 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2c92c9-a703-41d0-90c6-9cdda5555cf7 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:31.614432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2c92c9-a703-41d0-90c6-9cdda5555cf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:31.698377 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-af2c92c9-a703-41d0-90c6-9cdda5555cf7 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:31.703718 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-af2c92c9-a703-41d0-90c6-9cdda5555cf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:31.989435 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 838/1709] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:32.007413 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1e0eae-c0ad-45fe-8f65-20f85228ab8f admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:32.010953 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-db1e0eae-c0ad-45fe-8f65-20f85228ab8f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:32.102181 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1e0eae-c0ad-45fe-8f65-20f85228ab8f admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:32.106986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be34d3e4-a034-4db8-9bfd-3d323902857a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:32.107527 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be34d3e4-a034-4db8-9bfd-3d323902857a service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:32.107841 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be34d3e4-a034-4db8-9bfd-3d323902857a service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:32.108135 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be34d3e4-a034-4db8-9bfd-3d323902857a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:32.121188 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be34d3e4-a034-4db8-9bfd-3d323902857a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:32.122439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be34d3e4-a034-4db8-9bfd-3d323902857a service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:32.133180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 872/1710] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:32.447284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2c92c9-a703-41d0-90c6-9cdda5555cf7 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:32.449661 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-af2c92c9-a703-41d0-90c6-9cdda5555cf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:32.475883 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2c92c9-a703-41d0-90c6-9cdda5555cf7 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:32.484145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311e02fa-bd00-49ad-b82a-642780a076ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:32.485397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311e02fa-bd00-49ad-b82a-642780a076ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:32.485899 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311e02fa-bd00-49ad-b82a-642780a076ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:32.486305 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311e02fa-bd00-49ad-b82a-642780a076ad admin admin] PATH_INFO: `/v3/projects/1fbf3a4d437f48a0bab6f7d5ed18d307` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:32.487670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311e02fa-bd00-49ad-b82a-642780a076ad admin admin] RBAC: Authorizing `identity:delete_project(project_id=1fbf3a4d437f48a0bab6f7d5ed18d307)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:32.490227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311e02fa-bd00-49ad-b82a-642780a076ad admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:32.551739 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-311e02fa-bd00-49ad-b82a-642780a076ad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1fbf3a4d437f48a0bab6f7d5ed18d307', 'request_id': 'req-311e02fa-bd00-49ad-b82a-642780a076ad'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:32.561824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-311e02fa-bd00-49ad-b82a-642780a076ad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1fbf3a4d437f48a0bab6f7d5ed18d307', 'request_id': 'req-311e02fa-bd00-49ad-b82a-642780a076ad'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:32.570716 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-311e02fa-bd00-49ad-b82a-642780a076ad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1fbf3a4d437f48a0bab6f7d5ed18d307', 'request_id': 'req-311e02fa-bd00-49ad-b82a-642780a076ad'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:32.594146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-311e02fa-bd00-49ad-b82a-642780a076ad admin admin] The token cache is being invalidate because project 1fbf3a4d437f48a0bab6f7d5ed18d307 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:32.594552 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-311e02fa-bd00-49ad-b82a-642780a076ad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:32.595891 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-311e02fa-bd00-49ad-b82a-642780a076ad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:32.598130 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 839/1711] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:32 2022] DELETE /identity/v3/projects/1fbf3a4d437f48a0bab6f7d5ed18d307 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:32.602197 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be34d3e4-a034-4db8-9bfd-3d323902857a service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:32.615828 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be34d3e4-a034-4db8-9bfd-3d323902857a service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:32.707830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59449ed-836b-43fc-84d2-ed0b709c000e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:32.708511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59449ed-836b-43fc-84d2-ed0b709c000e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:32.708795 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59449ed-836b-43fc-84d2-ed0b709c000e admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:32.709069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59449ed-836b-43fc-84d2-ed0b709c000e admin admin] PATH_INFO: `/v3/users/3bce99e644e4411e98b6b51b4f8a1fea` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:32.711462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59449ed-836b-43fc-84d2-ed0b709c000e admin admin] RBAC: Authorizing `identity:delete_user(user_id=3bce99e644e4411e98b6b51b4f8a1fea)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:32.714025 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59449ed-836b-43fc-84d2-ed0b709c000e admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:32.781786 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:32.829033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d59449ed-836b-43fc-84d2-ed0b709c000e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3bce99e644e4411e98b6b51b4f8a1fea'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:32.830052 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d59449ed-836b-43fc-84d2-ed0b709c000e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3bce99e644e4411e98b6b51b4f8a1fea'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:32.840186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d59449ed-836b-43fc-84d2-ed0b709c000e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3bce99e644e4411e98b6b51b4f8a1fea'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:32.861000 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d59449ed-836b-43fc-84d2-ed0b709c000e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3bce99e644e4411e98b6b51b4f8a1fea'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:32.870852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d59449ed-836b-43fc-84d2-ed0b709c000e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3bce99e644e4411e98b6b51b4f8a1fea'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:32.879059 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 873/1712] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:32 2022] DELETE /identity/v3/users/3bce99e644e4411e98b6b51b4f8a1fea => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:33.488289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311e02fa-bd00-49ad-b82a-642780a076ad admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:33.495816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-311e02fa-bd00-49ad-b82a-642780a076ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:33.521829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311e02fa-bd00-49ad-b82a-642780a076ad admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:33.628831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:33.661408 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59c5d03-74c8-4112-b899-93930fc05069 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:33.663334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59c5d03-74c8-4112-b899-93930fc05069 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:33.663955 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59c5d03-74c8-4112-b899-93930fc05069 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:33.664429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59c5d03-74c8-4112-b899-93930fc05069 admin admin] PATH_INFO: `/v3/projects/9eca102f1d9749a3a2723f07db30f289` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:33.666347 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59c5d03-74c8-4112-b899-93930fc05069 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9eca102f1d9749a3a2723f07db30f289)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:33.669896 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59c5d03-74c8-4112-b899-93930fc05069 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:33.844262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a59c5d03-74c8-4112-b899-93930fc05069 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9eca102f1d9749a3a2723f07db30f289', 'request_id': 'req-a59c5d03-74c8-4112-b899-93930fc05069'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:33.873481 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a59c5d03-74c8-4112-b899-93930fc05069 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9eca102f1d9749a3a2723f07db30f289', 'request_id': 'req-a59c5d03-74c8-4112-b899-93930fc05069'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:33.889145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a59c5d03-74c8-4112-b899-93930fc05069 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9eca102f1d9749a3a2723f07db30f289', 'request_id': 'req-a59c5d03-74c8-4112-b899-93930fc05069'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:33.934226 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a59c5d03-74c8-4112-b899-93930fc05069 admin admin] The token cache is being invalidate because project 9eca102f1d9749a3a2723f07db30f289 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:33.934488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a59c5d03-74c8-4112-b899-93930fc05069 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:33.935755 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a59c5d03-74c8-4112-b899-93930fc05069 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:33.943180 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 840/1713] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:33 2022] DELETE /identity/v3/projects/9eca102f1d9749a3a2723f07db30f289 => generated 0 bytes in 452 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:33.944741 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9747ba20-5471-43da-ae46-bd696f67e1d5 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:33.945236 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9747ba20-5471-43da-ae46-bd696f67e1d5 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:33.945349 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9747ba20-5471-43da-ae46-bd696f67e1d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:33.975164 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-9747ba20-5471-43da-ae46-bd696f67e1d5 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:33.976375 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-9747ba20-5471-43da-ae46-bd696f67e1d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:34.040767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 874/1714] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:34.048761 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13936b1-0f32-4389-98fe-9e33bcd416dd None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:34.049058 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13936b1-0f32-4389-98fe-9e33bcd416dd None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:34.049343 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13936b1-0f32-4389-98fe-9e33bcd416dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:34.069447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d13936b1-0f32-4389-98fe-9e33bcd416dd None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:34.070663 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d13936b1-0f32-4389-98fe-9e33bcd416dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:34.079497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 841/1715] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:34.083833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9747ba20-5471-43da-ae46-bd696f67e1d5 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:34.096032 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9747ba20-5471-43da-ae46-bd696f67e1d5 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:34.100950 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d129e584-38bb-4748-97df-4ed212379930 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:34.101552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d129e584-38bb-4748-97df-4ed212379930 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:34.101832 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d129e584-38bb-4748-97df-4ed212379930 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:34.102103 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d129e584-38bb-4748-97df-4ed212379930 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:34.102805 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d129e584-38bb-4748-97df-4ed212379930 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:58:34.103138 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d129e584-38bb-4748-97df-4ed212379930 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:34.105318 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d129e584-38bb-4748-97df-4ed212379930 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:34.116947 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 875/1716] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:58:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:58:34.121885 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13936b1-0f32-4389-98fe-9e33bcd416dd None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:34.131021 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13936b1-0f32-4389-98fe-9e33bcd416dd None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:34.136903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8354c8a-0f85-4892-95d0-f90b6cd94a2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:34.137501 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8354c8a-0f85-4892-95d0-f90b6cd94a2f admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:34.137774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8354c8a-0f85-4892-95d0-f90b6cd94a2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:34.138042 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8354c8a-0f85-4892-95d0-f90b6cd94a2f admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:34.138699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8354c8a-0f85-4892-95d0-f90b6cd94a2f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:34.140611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8354c8a-0f85-4892-95d0-f90b6cd94a2f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:34.168145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 842/1717] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:34 2022] POST /identity/v3/projects => generated 367 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:34.172372 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d129e584-38bb-4748-97df-4ed212379930 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:34.183958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d129e584-38bb-4748-97df-4ed212379930 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:34.248944 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44639a12-b1ac-4563-9c6b-c4ce09402953 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:34.249585 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44639a12-b1ac-4563-9c6b-c4ce09402953 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:34.249867 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44639a12-b1ac-4563-9c6b-c4ce09402953 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:34.250141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44639a12-b1ac-4563-9c6b-c4ce09402953 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:34.250793 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44639a12-b1ac-4563-9c6b-c4ce09402953 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:34.252815 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44639a12-b1ac-4563-9c6b-c4ce09402953 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:34.275243 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 876/1718] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:34 2022] POST /identity/v3/users => generated 284 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:34.280127 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8354c8a-0f85-4892-95d0-f90b6cd94a2f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:34.290683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8354c8a-0f85-4892-95d0-f90b6cd94a2f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:34.296050 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8efa33-f340-443a-84b7-5920e542c27f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:34.296935 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8efa33-f340-443a-84b7-5920e542c27f admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:34.297210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8efa33-f340-443a-84b7-5920e542c27f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:34.297480 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8efa33-f340-443a-84b7-5920e542c27f admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:34.298117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8efa33-f340-443a-84b7-5920e542c27f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:34.299755 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8efa33-f340-443a-84b7-5920e542c27f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:34.310756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 843/1719] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:34 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:34.315059 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44639a12-b1ac-4563-9c6b-c4ce09402953 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:34.326135 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44639a12-b1ac-4563-9c6b-c4ce09402953 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:34.330930 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b50ed5-c622-4041-8f2e-bed67763d224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:34.331547 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b50ed5-c622-4041-8f2e-bed67763d224 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:34.331860 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b50ed5-c622-4041-8f2e-bed67763d224 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:34.332137 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b50ed5-c622-4041-8f2e-bed67763d224 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:34.332789 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b50ed5-c622-4041-8f2e-bed67763d224 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:34.334405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b50ed5-c622-4041-8f2e-bed67763d224 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:34.346791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 877/1720] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:34 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:34.351596 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8efa33-f340-443a-84b7-5920e542c27f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:34.363230 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8efa33-f340-443a-84b7-5920e542c27f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:34.368001 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c84ca9f-2fd2-413a-90f8-9de4e3a3a950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:34.368669 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c84ca9f-2fd2-413a-90f8-9de4e3a3a950 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:34.368955 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c84ca9f-2fd2-413a-90f8-9de4e3a3a950 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:34.369268 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c84ca9f-2fd2-413a-90f8-9de4e3a3a950 admin admin] PATH_INFO: `/v3/projects/809f1bfaf9db45ccaa61dd5d5aa0cee7/users/54ce521495334137ba178ec2e053197e/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:34.385279 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c84ca9f-2fd2-413a-90f8-9de4e3a3a950 admin admin] RBAC: Authorizing `identity:create_grant(project_id=809f1bfaf9db45ccaa61dd5d5aa0cee7, user_id=54ce521495334137ba178ec2e053197e, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:34.388049 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c84ca9f-2fd2-413a-90f8-9de4e3a3a950 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:34.404365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 844/1721] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:34 2022] PUT /identity/v3/projects/809f1bfaf9db45ccaa61dd5d5aa0cee7/users/54ce521495334137ba178ec2e053197e/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:58:34.415839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2050d3e4-b05e-44c8-83d3-fa19531e9776 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:34.416170 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2050d3e4-b05e-44c8-83d3-fa19531e9776 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:34.416944 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2050d3e4-b05e-44c8-83d3-fa19531e9776 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:34.424643 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:DELETE=8,keystone:SELECT=88,keystone:INSERT=4 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:34.449754 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-2050d3e4-b05e-44c8-83d3-fa19531e9776 None None] MFA Rules not processed for user `54ce521495334137ba178ec2e053197e`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:34.451280 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-2050d3e4-b05e-44c8-83d3-fa19531e9776 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:34.537138 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 878/1722] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:34 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:34.542574 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c84ca9f-2fd2-413a-90f8-9de4e3a3a950 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:34.623296 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c84ca9f-2fd2-413a-90f8-9de4e3a3a950 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:34.628019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89103c73-693c-46f0-b347-fe139470a234 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:34.628650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89103c73-693c-46f0-b347-fe139470a234 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:34.628926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89103c73-693c-46f0-b347-fe139470a234 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:34.629201 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89103c73-693c-46f0-b347-fe139470a234 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:34.630397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89103c73-693c-46f0-b347-fe139470a234 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:34.631854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89103c73-693c-46f0-b347-fe139470a234 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:34.649013 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:34.668891 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 845/1723] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:34 2022] POST /identity/v3/projects => generated 367 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:34.674368 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2050d3e4-b05e-44c8-83d3-fa19531e9776 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:34.694078 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2050d3e4-b05e-44c8-83d3-fa19531e9776 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:34.797244 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1477241-5d1f-4762-a9f8-986a922935f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:34.797877 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1477241-5d1f-4762-a9f8-986a922935f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:34.798155 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1477241-5d1f-4762-a9f8-986a922935f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:34.798425 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1477241-5d1f-4762-a9f8-986a922935f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:34.799091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1477241-5d1f-4762-a9f8-986a922935f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:34.801034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1477241-5d1f-4762-a9f8-986a922935f8 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:34.826708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 879/1724] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:34 2022] POST /identity/v3/users => generated 290 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:34.830758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89103c73-693c-46f0-b347-fe139470a234 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:34.842437 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89103c73-693c-46f0-b347-fe139470a234 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:34.847384 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fc0ab9-762d-40d2-bbde-7951b7034606 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:34.848156 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fc0ab9-762d-40d2-bbde-7951b7034606 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:34.848432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fc0ab9-762d-40d2-bbde-7951b7034606 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:34.848699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fc0ab9-762d-40d2-bbde-7951b7034606 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:34.849343 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08fc0ab9-762d-40d2-bbde-7951b7034606 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:34.850969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08fc0ab9-762d-40d2-bbde-7951b7034606 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:34.869134 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 846/1725] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:34 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:34.870283 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1477241-5d1f-4762-a9f8-986a922935f8 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:34.881427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1477241-5d1f-4762-a9f8-986a922935f8 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:34.886201 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca72399d-4d0a-47d8-a93e-eaaffb8f56c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:34.886906 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca72399d-4d0a-47d8-a93e-eaaffb8f56c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:34.887192 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca72399d-4d0a-47d8-a93e-eaaffb8f56c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:34.887470 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca72399d-4d0a-47d8-a93e-eaaffb8f56c4 admin admin] PATH_INFO: `/v3/projects/d4191f6b320a49b38e96260733c96dfd/users/43fe577eb78a41aeb1b53d723c4e523e/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:34.906752 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca72399d-4d0a-47d8-a93e-eaaffb8f56c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4191f6b320a49b38e96260733c96dfd, user_id=43fe577eb78a41aeb1b53d723c4e523e, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:34.909508 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca72399d-4d0a-47d8-a93e-eaaffb8f56c4 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:34.925810 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 880/1726] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:34 2022] PUT /identity/v3/projects/d4191f6b320a49b38e96260733c96dfd/users/43fe577eb78a41aeb1b53d723c4e523e/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:58:34.930131 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fc0ab9-762d-40d2-bbde-7951b7034606 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:34.944524 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fc0ab9-762d-40d2-bbde-7951b7034606 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:35.035866 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621f1166-c2d6-4244-aa86-79e70a087247 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:35.036607 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621f1166-c2d6-4244-aa86-79e70a087247 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:35.036988 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621f1166-c2d6-4244-aa86-79e70a087247 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:35.037380 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621f1166-c2d6-4244-aa86-79e70a087247 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:35.038333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621f1166-c2d6-4244-aa86-79e70a087247 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:35.040634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621f1166-c2d6-4244-aa86-79e70a087247 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:35.059312 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 847/1727] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:34 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:35.063641 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca72399d-4d0a-47d8-a93e-eaaffb8f56c4 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:35.078336 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca72399d-4d0a-47d8-a93e-eaaffb8f56c4 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:35.084130 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a72bd16-efd1-4c9c-9410-4adc0da7f1fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:35.085077 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a72bd16-efd1-4c9c-9410-4adc0da7f1fb admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:35.085358 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a72bd16-efd1-4c9c-9410-4adc0da7f1fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:35.085631 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a72bd16-efd1-4c9c-9410-4adc0da7f1fb admin admin] PATH_INFO: `/v3/projects/d4191f6b320a49b38e96260733c96dfd/users/43fe577eb78a41aeb1b53d723c4e523e/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:35.087646 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a72bd16-efd1-4c9c-9410-4adc0da7f1fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4191f6b320a49b38e96260733c96dfd, user_id=43fe577eb78a41aeb1b53d723c4e523e, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:35.091001 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a72bd16-efd1-4c9c-9410-4adc0da7f1fb admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:35.106050 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 881/1728] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:35 2022] PUT /identity/v3/projects/d4191f6b320a49b38e96260733c96dfd/users/43fe577eb78a41aeb1b53d723c4e523e/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:35.116112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f846345a-e43c-4b42-859f-ec9babaf78ce None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:35.116404 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f846345a-e43c-4b42-859f-ec9babaf78ce None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:35.116703 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f846345a-e43c-4b42-859f-ec9babaf78ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:35.152869 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f846345a-e43c-4b42-859f-ec9babaf78ce None None] MFA Rules not processed for user `43fe577eb78a41aeb1b53d723c4e523e`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:35.153594 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f846345a-e43c-4b42-859f-ec9babaf78ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:35.320058 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 848/1729] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:35 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:35.335149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a72bd16-efd1-4c9c-9410-4adc0da7f1fb admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:35.337344 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6a72bd16-efd1-4c9c-9410-4adc0da7f1fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:35.421728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:35.439661 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a72bd16-efd1-4c9c-9410-4adc0da7f1fb admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:35.445546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527b789a-937c-41b4-a0d7-1314c55108f5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:35.445546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527b789a-937c-41b4-a0d7-1314c55108f5 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:35.445546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527b789a-937c-41b4-a0d7-1314c55108f5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:35.445546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527b789a-937c-41b4-a0d7-1314c55108f5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:35.457607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527b789a-937c-41b4-a0d7-1314c55108f5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:35.459178 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527b789a-937c-41b4-a0d7-1314c55108f5 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:35.470691 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 882/1730] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:35 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:35.568953 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f846345a-e43c-4b42-859f-ec9babaf78ce None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:35.580959 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f846345a-e43c-4b42-859f-ec9babaf78ce None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:35.584847 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d834b0-0826-4958-a8c1-97cd0aeb54ba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:35.585470 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d834b0-0826-4958-a8c1-97cd0aeb54ba service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:35.585752 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d834b0-0826-4958-a8c1-97cd0aeb54ba service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:35.586022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d834b0-0826-4958-a8c1-97cd0aeb54ba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:35.664670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d834b0-0826-4958-a8c1-97cd0aeb54ba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:35.666142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d834b0-0826-4958-a8c1-97cd0aeb54ba service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:35.671877 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:35.680279 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 849/1731] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:35 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:37.920253 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527b789a-937c-41b4-a0d7-1314c55108f5 service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:37.922925 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-527b789a-937c-41b4-a0d7-1314c55108f5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:38.045129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527b789a-937c-41b4-a0d7-1314c55108f5 service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:38.065532 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fc7f80-ecd2-4940-99b5-bdfce4f8f3b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:38.066472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fc7f80-ecd2-4940-99b5-bdfce4f8f3b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:38.066589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fc7f80-ecd2-4940-99b5-bdfce4f8f3b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:38.066885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fc7f80-ecd2-4940-99b5-bdfce4f8f3b2 admin admin] PATH_INFO: `/v3/users/3fad6c6ca4614270856c689c5bb46a5f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:38.068732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fc7f80-ecd2-4940-99b5-bdfce4f8f3b2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3fad6c6ca4614270856c689c5bb46a5f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:38.070904 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fc7f80-ecd2-4940-99b5-bdfce4f8f3b2 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:38.197390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-23fc7f80-ecd2-4940-99b5-bdfce4f8f3b2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3fad6c6ca4614270856c689c5bb46a5f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:38.198328 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-23fc7f80-ecd2-4940-99b5-bdfce4f8f3b2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3fad6c6ca4614270856c689c5bb46a5f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:38.226583 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-23fc7f80-ecd2-4940-99b5-bdfce4f8f3b2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3fad6c6ca4614270856c689c5bb46a5f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:38.272778 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-23fc7f80-ecd2-4940-99b5-bdfce4f8f3b2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3fad6c6ca4614270856c689c5bb46a5f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:38.293799 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-23fc7f80-ecd2-4940-99b5-bdfce4f8f3b2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3fad6c6ca4614270856c689c5bb46a5f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:38.303938 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 883/1732] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:37 2022] DELETE /identity/v3/users/3fad6c6ca4614270856c689c5bb46a5f => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:38.632406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d834b0-0826-4958-a8c1-97cd0aeb54ba service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:38.648103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d834b0-0826-4958-a8c1-97cd0aeb54ba service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:38.743174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c96b38-9465-4db0-848b-47760698754f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:38.743923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c96b38-9465-4db0-848b-47760698754f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:38.744299 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c96b38-9465-4db0-848b-47760698754f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:38.745266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c96b38-9465-4db0-848b-47760698754f admin admin] PATH_INFO: `/v3/projects/a850973394834b1c9a2eb5370e303a21` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:38.749234 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c96b38-9465-4db0-848b-47760698754f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a850973394834b1c9a2eb5370e303a21)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:38.753913 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c96b38-9465-4db0-848b-47760698754f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:38.820755 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e6c96b38-9465-4db0-848b-47760698754f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a850973394834b1c9a2eb5370e303a21', 'request_id': 'req-e6c96b38-9465-4db0-848b-47760698754f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:38.833027 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e6c96b38-9465-4db0-848b-47760698754f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a850973394834b1c9a2eb5370e303a21', 'request_id': 'req-e6c96b38-9465-4db0-848b-47760698754f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:38.845034 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e6c96b38-9465-4db0-848b-47760698754f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a850973394834b1c9a2eb5370e303a21', 'request_id': 'req-e6c96b38-9465-4db0-848b-47760698754f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:38.875944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e6c96b38-9465-4db0-848b-47760698754f admin admin] The token cache is being invalidate because project a850973394834b1c9a2eb5370e303a21 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:38.876627 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e6c96b38-9465-4db0-848b-47760698754f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:38.877752 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e6c96b38-9465-4db0-848b-47760698754f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:38.880217 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 850/1733] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:38 2022] DELETE /identity/v3/projects/a850973394834b1c9a2eb5370e303a21 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:38.883967 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fc7f80-ecd2-4940-99b5-bdfce4f8f3b2 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:38.886462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-23fc7f80-ecd2-4940-99b5-bdfce4f8f3b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:38.907761 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fc7f80-ecd2-4940-99b5-bdfce4f8f3b2 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:39.005020 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ebb478-48c2-4d35-9dc2-e9da6b776a6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:39.005848 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ebb478-48c2-4d35-9dc2-e9da6b776a6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:39.006238 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ebb478-48c2-4d35-9dc2-e9da6b776a6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:39.006619 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ebb478-48c2-4d35-9dc2-e9da6b776a6e admin admin] PATH_INFO: `/v3/users/201e23b6153b41c5b928747f417c0774` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:39.009783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ebb478-48c2-4d35-9dc2-e9da6b776a6e admin admin] RBAC: Authorizing `identity:delete_user(user_id=201e23b6153b41c5b928747f417c0774)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:39.014557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ebb478-48c2-4d35-9dc2-e9da6b776a6e admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:39.025936 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:39.112606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-30ebb478-48c2-4d35-9dc2-e9da6b776a6e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '201e23b6153b41c5b928747f417c0774'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:39.113468 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-30ebb478-48c2-4d35-9dc2-e9da6b776a6e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '201e23b6153b41c5b928747f417c0774'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:39.130645 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-30ebb478-48c2-4d35-9dc2-e9da6b776a6e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '201e23b6153b41c5b928747f417c0774'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:39.163390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-30ebb478-48c2-4d35-9dc2-e9da6b776a6e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '201e23b6153b41c5b928747f417c0774'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:39.180870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-30ebb478-48c2-4d35-9dc2-e9da6b776a6e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '201e23b6153b41c5b928747f417c0774'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:39.206887 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 884/1734] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:38 2022] DELETE /identity/v3/users/201e23b6153b41c5b928747f417c0774 => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:58:39.692451 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c96b38-9465-4db0-848b-47760698754f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:39.693957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e6c96b38-9465-4db0-848b-47760698754f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:39.716528 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c96b38-9465-4db0-848b-47760698754f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:39.824014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afaebd2a-45db-4583-ba5e-d6791b2dc4d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:39.824673 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaebd2a-45db-4583-ba5e-d6791b2dc4d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:39.825125 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaebd2a-45db-4583-ba5e-d6791b2dc4d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:39.825394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaebd2a-45db-4583-ba5e-d6791b2dc4d8 admin admin] PATH_INFO: `/v3/projects/94e12a99f9f541e5a3b2b643ab0c0332` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:39.826820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afaebd2a-45db-4583-ba5e-d6791b2dc4d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=94e12a99f9f541e5a3b2b643ab0c0332)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:39.829176 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afaebd2a-45db-4583-ba5e-d6791b2dc4d8 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:39.888592 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:39.920696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-afaebd2a-45db-4583-ba5e-d6791b2dc4d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '94e12a99f9f541e5a3b2b643ab0c0332', 'request_id': 'req-afaebd2a-45db-4583-ba5e-d6791b2dc4d8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:39.938686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-afaebd2a-45db-4583-ba5e-d6791b2dc4d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '94e12a99f9f541e5a3b2b643ab0c0332', 'request_id': 'req-afaebd2a-45db-4583-ba5e-d6791b2dc4d8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:39.948835 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-afaebd2a-45db-4583-ba5e-d6791b2dc4d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '94e12a99f9f541e5a3b2b643ab0c0332', 'request_id': 'req-afaebd2a-45db-4583-ba5e-d6791b2dc4d8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:39.963837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-afaebd2a-45db-4583-ba5e-d6791b2dc4d8 admin admin] The token cache is being invalidate because project 94e12a99f9f541e5a3b2b643ab0c0332 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:39.964210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-afaebd2a-45db-4583-ba5e-d6791b2dc4d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:39.965029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-afaebd2a-45db-4583-ba5e-d6791b2dc4d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:39.969980 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 851/1735] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:39 2022] DELETE /identity/v3/projects/94e12a99f9f541e5a3b2b643ab0c0332 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:39.971740 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffb1095-fbfa-4e22-9835-5ddda1ad4794 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:39.972038 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffb1095-fbfa-4e22-9835-5ddda1ad4794 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:39.972313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffb1095-fbfa-4e22-9835-5ddda1ad4794 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:39.997445 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-8ffb1095-fbfa-4e22-9835-5ddda1ad4794 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:40.002241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8ffb1095-fbfa-4e22-9835-5ddda1ad4794 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:40.147769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 885/1736] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:40.159349 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d36eb8e-c39a-4323-9f80-39983b9510eb None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:40.159790 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d36eb8e-c39a-4323-9f80-39983b9510eb None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:40.160044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d36eb8e-c39a-4323-9f80-39983b9510eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:40.219575 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-7d36eb8e-c39a-4323-9f80-39983b9510eb None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:40.228048 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7d36eb8e-c39a-4323-9f80-39983b9510eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:40.250658 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 852/1737] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:40.267575 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffb1095-fbfa-4e22-9835-5ddda1ad4794 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:40.282301 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffb1095-fbfa-4e22-9835-5ddda1ad4794 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:40.289279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eca9340-8fa4-4ae9-ba6c-5c02f05d7bd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:40.289963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eca9340-8fa4-4ae9-ba6c-5c02f05d7bd3 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:40.290243 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eca9340-8fa4-4ae9-ba6c-5c02f05d7bd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:40.290691 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eca9340-8fa4-4ae9-ba6c-5c02f05d7bd3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:40.291581 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eca9340-8fa4-4ae9-ba6c-5c02f05d7bd3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:58:40.292469 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eca9340-8fa4-4ae9-ba6c-5c02f05d7bd3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:40.296228 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eca9340-8fa4-4ae9-ba6c-5c02f05d7bd3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:40.336682 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 886/1738] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:58:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:58:40.342552 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d36eb8e-c39a-4323-9f80-39983b9510eb None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:40.355091 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d36eb8e-c39a-4323-9f80-39983b9510eb None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:40.361147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bc17e5-4fc7-4a36-b294-04ea2eb0651c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:40.361929 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bc17e5-4fc7-4a36-b294-04ea2eb0651c admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:40.362106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bc17e5-4fc7-4a36-b294-04ea2eb0651c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:40.362351 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bc17e5-4fc7-4a36-b294-04ea2eb0651c admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:40.363029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bc17e5-4fc7-4a36-b294-04ea2eb0651c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:40.364976 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bc17e5-4fc7-4a36-b294-04ea2eb0651c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:40.407670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 853/1739] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:40 2022] POST /identity/v3/projects => generated 373 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:40.412228 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eca9340-8fa4-4ae9-ba6c-5c02f05d7bd3 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:40.424677 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eca9340-8fa4-4ae9-ba6c-5c02f05d7bd3 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:40.520075 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd65ba3-a2e3-450d-8c71-ca9452af4e39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:40.520966 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd65ba3-a2e3-450d-8c71-ca9452af4e39 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:40.521421 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd65ba3-a2e3-450d-8c71-ca9452af4e39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:40.521847 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd65ba3-a2e3-450d-8c71-ca9452af4e39 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:40.522782 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd65ba3-a2e3-450d-8c71-ca9452af4e39 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:40.524969 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd65ba3-a2e3-450d-8c71-ca9452af4e39 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:40.550977 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 887/1740] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:40 2022] POST /identity/v3/users => generated 287 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:58:40.555283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bc17e5-4fc7-4a36-b294-04ea2eb0651c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:40.579867 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bc17e5-4fc7-4a36-b294-04ea2eb0651c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:40.586380 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c90188-51bc-43ee-bd1b-4d83a4b245d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:40.587377 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c90188-51bc-43ee-bd1b-4d83a4b245d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:40.587662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c90188-51bc-43ee-bd1b-4d83a4b245d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:40.587958 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c90188-51bc-43ee-bd1b-4d83a4b245d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:40.589244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c90188-51bc-43ee-bd1b-4d83a4b245d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:40.592327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c90188-51bc-43ee-bd1b-4d83a4b245d0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:40.605709 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 854/1741] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:40 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 00:58:40.609361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd65ba3-a2e3-450d-8c71-ca9452af4e39 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:40.616972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:40.624036 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd65ba3-a2e3-450d-8c71-ca9452af4e39 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:40.632536 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd241a9e-96ba-446c-8eef-b1ef6bbe19c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:40.634083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd241a9e-96ba-446c-8eef-b1ef6bbe19c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:40.636974 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd241a9e-96ba-446c-8eef-b1ef6bbe19c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:40.638140 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd241a9e-96ba-446c-8eef-b1ef6bbe19c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:40.639958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd241a9e-96ba-446c-8eef-b1ef6bbe19c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:40.644615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd241a9e-96ba-446c-8eef-b1ef6bbe19c7 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:40.660767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 888/1742] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:40 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:40.665047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c90188-51bc-43ee-bd1b-4d83a4b245d0 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:40.680478 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c90188-51bc-43ee-bd1b-4d83a4b245d0 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:40.687020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8969154-3df6-4576-b9ce-99f493638e64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:40.687720 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8969154-3df6-4576-b9ce-99f493638e64 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:40.687986 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8969154-3df6-4576-b9ce-99f493638e64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:40.688270 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8969154-3df6-4576-b9ce-99f493638e64 admin admin] PATH_INFO: `/v3/projects/217097ae31cf4319b2c373e5fc625d91/users/70806ccd2140449a9a54b3ec7b65fc85/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:40.694956 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd241a9e-96ba-446c-8eef-b1ef6bbe19c7 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:40.697078 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-fd241a9e-96ba-446c-8eef-b1ef6bbe19c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:40.719356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd241a9e-96ba-446c-8eef-b1ef6bbe19c7 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:40.722173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8969154-3df6-4576-b9ce-99f493638e64 admin admin] RBAC: Authorizing `identity:create_grant(project_id=217097ae31cf4319b2c373e5fc625d91, user_id=70806ccd2140449a9a54b3ec7b65fc85, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:40.725074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8969154-3df6-4576-b9ce-99f493638e64 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:40.726133 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb59f195-8805-4bf9-bdbd-e8ca148e03a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:40.726802 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb59f195-8805-4bf9-bdbd-e8ca148e03a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:40.727295 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb59f195-8805-4bf9-bdbd-e8ca148e03a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:40.728205 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb59f195-8805-4bf9-bdbd-e8ca148e03a9 admin admin] PATH_INFO: `/v3/users/3aa7409c9f6f46f78164818b6f6927e4` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:40.730704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb59f195-8805-4bf9-bdbd-e8ca148e03a9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3aa7409c9f6f46f78164818b6f6927e4)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:40.734310 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb59f195-8805-4bf9-bdbd-e8ca148e03a9 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:40.749037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 855/1743] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:40 2022] PUT /identity/v3/projects/217097ae31cf4319b2c373e5fc625d91/users/70806ccd2140449a9a54b3ec7b65fc85/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:40.761857 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d8232f-29a4-4a9c-8ae2-a3906b4ee7ca None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:40.762295 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d8232f-29a4-4a9c-8ae2-a3906b4ee7ca None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:40.762574 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d8232f-29a4-4a9c-8ae2-a3906b4ee7ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:40.799854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-fb59f195-8805-4bf9-bdbd-e8ca148e03a9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3aa7409c9f6f46f78164818b6f6927e4'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:40.800536 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-fb59f195-8805-4bf9-bdbd-e8ca148e03a9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3aa7409c9f6f46f78164818b6f6927e4'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:40.800685 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b8d8232f-29a4-4a9c-8ae2-a3906b4ee7ca None None] MFA Rules not processed for user `70806ccd2140449a9a54b3ec7b65fc85`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:40.801830 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b8d8232f-29a4-4a9c-8ae2-a3906b4ee7ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:40.811177 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-fb59f195-8805-4bf9-bdbd-e8ca148e03a9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3aa7409c9f6f46f78164818b6f6927e4'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:40.827912 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-fb59f195-8805-4bf9-bdbd-e8ca148e03a9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3aa7409c9f6f46f78164818b6f6927e4'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:40.836476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-fb59f195-8805-4bf9-bdbd-e8ca148e03a9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3aa7409c9f6f46f78164818b6f6927e4'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:40.845068 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 889/1744] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:40 2022] DELETE /identity/v3/users/3aa7409c9f6f46f78164818b6f6927e4 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:40.890683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 856/1745] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:40 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:40.904841 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb59f195-8805-4bf9-bdbd-e8ca148e03a9 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:40.906320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-fb59f195-8805-4bf9-bdbd-e8ca148e03a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:40.968264 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb59f195-8805-4bf9-bdbd-e8ca148e03a9 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:40.972321 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1661a3-e43f-404a-802b-2ee0ea682856 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:40.972941 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1661a3-e43f-404a-802b-2ee0ea682856 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:40.973220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1661a3-e43f-404a-802b-2ee0ea682856 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:40.973492 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1661a3-e43f-404a-802b-2ee0ea682856 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:41.035849 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d8232f-29a4-4a9c-8ae2-a3906b4ee7ca None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:41.037389 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b8d8232f-29a4-4a9c-8ae2-a3906b4ee7ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:41.040289 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1661a3-e43f-404a-802b-2ee0ea682856 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:41.041970 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1661a3-e43f-404a-802b-2ee0ea682856 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:41.051670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 890/1746] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:40 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:41.076800 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1661a3-e43f-404a-802b-2ee0ea682856 service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:41.078344 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4c1661a3-e43f-404a-802b-2ee0ea682856 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:41.108913 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:41.114213 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:41.143270 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d8232f-29a4-4a9c-8ae2-a3906b4ee7ca None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:41.185213 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1661a3-e43f-404a-802b-2ee0ea682856 service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:41.191192 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb60f14a-5f3d-4b67-b9ce-99af29ae690a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:41.191928 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb60f14a-5f3d-4b67-b9ce-99af29ae690a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:41.192391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb60f14a-5f3d-4b67-b9ce-99af29ae690a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:41.192560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb60f14a-5f3d-4b67-b9ce-99af29ae690a admin admin] PATH_INFO: `/v3/users/70806ccd2140449a9a54b3ec7b65fc85` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:41.194285 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb60f14a-5f3d-4b67-b9ce-99af29ae690a admin admin] RBAC: Authorizing `identity:delete_user(user_id=70806ccd2140449a9a54b3ec7b65fc85)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:41.196396 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb60f14a-5f3d-4b67-b9ce-99af29ae690a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:41.237621 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d64126-20d6-4f9a-90cd-d3c26f32f9ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:41.238378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d64126-20d6-4f9a-90cd-d3c26f32f9ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:41.238781 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d64126-20d6-4f9a-90cd-d3c26f32f9ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:41.239229 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d64126-20d6-4f9a-90cd-d3c26f32f9ec admin admin] PATH_INFO: `/v3/projects/c89fd0e945344feab91a37fdb93812e5` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:41.240742 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d64126-20d6-4f9a-90cd-d3c26f32f9ec admin admin] RBAC: Authorizing `identity:delete_project(project_id=c89fd0e945344feab91a37fdb93812e5)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:41.243314 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d64126-20d6-4f9a-90cd-d3c26f32f9ec admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:41.258588 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-fb60f14a-5f3d-4b67-b9ce-99af29ae690a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '70806ccd2140449a9a54b3ec7b65fc85'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:41.259313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-fb60f14a-5f3d-4b67-b9ce-99af29ae690a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '70806ccd2140449a9a54b3ec7b65fc85'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:41.285232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-fb60f14a-5f3d-4b67-b9ce-99af29ae690a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '70806ccd2140449a9a54b3ec7b65fc85'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:41.305775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-20d64126-20d6-4f9a-90cd-d3c26f32f9ec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c89fd0e945344feab91a37fdb93812e5', 'request_id': 'req-20d64126-20d6-4f9a-90cd-d3c26f32f9ec'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:41.308880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-fb60f14a-5f3d-4b67-b9ce-99af29ae690a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '70806ccd2140449a9a54b3ec7b65fc85'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:41.320190 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-fb60f14a-5f3d-4b67-b9ce-99af29ae690a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '70806ccd2140449a9a54b3ec7b65fc85'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:41.321478 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-20d64126-20d6-4f9a-90cd-d3c26f32f9ec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c89fd0e945344feab91a37fdb93812e5', 'request_id': 'req-20d64126-20d6-4f9a-90cd-d3c26f32f9ec'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:41.330382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 891/1747] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:41 2022] DELETE /identity/v3/users/70806ccd2140449a9a54b3ec7b65fc85 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:41.332030 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-20d64126-20d6-4f9a-90cd-d3c26f32f9ec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c89fd0e945344feab91a37fdb93812e5', 'request_id': 'req-20d64126-20d6-4f9a-90cd-d3c26f32f9ec'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:41.334918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bba50c-7ac0-4141-a4d8-e13513a9d130 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:41.335249 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bba50c-7ac0-4141-a4d8-e13513a9d130 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:41.335649 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bba50c-7ac0-4141-a4d8-e13513a9d130 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:41.366022 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-73bba50c-7ac0-4141-a4d8-e13513a9d130 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:41.367275 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-73bba50c-7ac0-4141-a4d8-e13513a9d130 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:41.372549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-20d64126-20d6-4f9a-90cd-d3c26f32f9ec admin admin] The token cache is being invalidate because project c89fd0e945344feab91a37fdb93812e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:41.372904 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-20d64126-20d6-4f9a-90cd-d3c26f32f9ec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:41.373676 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-20d64126-20d6-4f9a-90cd-d3c26f32f9ec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:41.375658 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 857/1748] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:41 2022] DELETE /identity/v3/projects/c89fd0e945344feab91a37fdb93812e5 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:58:41.382537 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af364e7f-6ad2-4d00-88a1-59c409b60fbe None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:41.382996 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af364e7f-6ad2-4d00-88a1-59c409b60fbe None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:41.383520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af364e7f-6ad2-4d00-88a1-59c409b60fbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:41.419841 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-af364e7f-6ad2-4d00-88a1-59c409b60fbe None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:41.421514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-af364e7f-6ad2-4d00-88a1-59c409b60fbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:41.480262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 892/1749] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:41.491154 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bba50c-7ac0-4141-a4d8-e13513a9d130 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:41.492350 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-73bba50c-7ac0-4141-a4d8-e13513a9d130 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:41.524328 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 858/1750] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:41.529283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af364e7f-6ad2-4d00-88a1-59c409b60fbe None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:41.530829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-af364e7f-6ad2-4d00-88a1-59c409b60fbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:41.547611 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:41.549525 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af364e7f-6ad2-4d00-88a1-59c409b60fbe None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:41.556403 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19433ad3-2495-4dc1-a369-4c2f4cb3d593 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:41.557787 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19433ad3-2495-4dc1-a369-4c2f4cb3d593 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:41.558247 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19433ad3-2495-4dc1-a369-4c2f4cb3d593 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:41.558612 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19433ad3-2495-4dc1-a369-4c2f4cb3d593 admin admin] PATH_INFO: `/v3/users/54ce521495334137ba178ec2e053197e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:41.560189 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19433ad3-2495-4dc1-a369-4c2f4cb3d593 admin admin] RBAC: Authorizing `identity:delete_user(user_id=54ce521495334137ba178ec2e053197e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:41.562890 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19433ad3-2495-4dc1-a369-4c2f4cb3d593 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:41.573790 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bba50c-7ac0-4141-a4d8-e13513a9d130 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:41.579150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8253d9-e5fb-4081-8cc5-99c372409efc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:41.579774 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8253d9-e5fb-4081-8cc5-99c372409efc service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:41.580202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8253d9-e5fb-4081-8cc5-99c372409efc service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:41.580503 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8253d9-e5fb-4081-8cc5-99c372409efc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:41.589027 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:41.593269 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8253d9-e5fb-4081-8cc5-99c372409efc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:41.594900 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8253d9-e5fb-4081-8cc5-99c372409efc service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:41.607475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 893/1751] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:41.609846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d946c4-b557-466d-8a8d-6bd5cd9b1e1b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:41.610343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d946c4-b557-466d-8a8d-6bd5cd9b1e1b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:41.610840 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d946c4-b557-466d-8a8d-6bd5cd9b1e1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:41.640154 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-b0d946c4-b557-466d-8a8d-6bd5cd9b1e1b None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:41.641484 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b0d946c4-b557-466d-8a8d-6bd5cd9b1e1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:41.641662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-19433ad3-2495-4dc1-a369-4c2f4cb3d593 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '54ce521495334137ba178ec2e053197e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:41.642452 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-19433ad3-2495-4dc1-a369-4c2f4cb3d593 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '54ce521495334137ba178ec2e053197e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:41.652505 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-19433ad3-2495-4dc1-a369-4c2f4cb3d593 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '54ce521495334137ba178ec2e053197e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:41.652665 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 894/1752] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:41.653767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-19433ad3-2495-4dc1-a369-4c2f4cb3d593 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '54ce521495334137ba178ec2e053197e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:41.657462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d946c4-b557-466d-8a8d-6bd5cd9b1e1b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:41.668780 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-19433ad3-2495-4dc1-a369-4c2f4cb3d593 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '54ce521495334137ba178ec2e053197e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:41.669508 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d946c4-b557-466d-8a8d-6bd5cd9b1e1b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:41.676614 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea86295-1841-4558-8e0d-75948223511a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:41.677427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea86295-1841-4558-8e0d-75948223511a admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:41.677702 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea86295-1841-4558-8e0d-75948223511a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:41.677976 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea86295-1841-4558-8e0d-75948223511a admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:41.678625 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea86295-1841-4558-8e0d-75948223511a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:58:41.678972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea86295-1841-4558-8e0d-75948223511a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:41.680589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea86295-1841-4558-8e0d-75948223511a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:41.689336 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 859/1753] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:41 2022] DELETE /identity/v3/users/54ce521495334137ba178ec2e053197e => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:41.691315 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f206791-2d53-4c10-b2ed-3b84e8612684 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:41.691789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f206791-2d53-4c10-b2ed-3b84e8612684 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:41.692383 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f206791-2d53-4c10-b2ed-3b84e8612684 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:41.699726 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 895/1754] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:58:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:58:41.701043 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea86295-1841-4558-8e0d-75948223511a admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:41.713459 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea86295-1841-4558-8e0d-75948223511a admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:41.717264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5f206791-2d53-4c10-b2ed-3b84e8612684 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:41.718495 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5f206791-2d53-4c10-b2ed-3b84e8612684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:41.798494 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 860/1755] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:41.805264 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e9a6ac-8e8c-4269-b31f-e979936efd5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:41.807386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e9a6ac-8e8c-4269-b31f-e979936efd5a admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:41.807828 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e9a6ac-8e8c-4269-b31f-e979936efd5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:41.808112 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e9a6ac-8e8c-4269-b31f-e979936efd5a admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:41.808764 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e9a6ac-8e8c-4269-b31f-e979936efd5a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:41.809136 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f206791-2d53-4c10-b2ed-3b84e8612684 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:41.810707 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e9a6ac-8e8c-4269-b31f-e979936efd5a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:41.811591 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5f206791-2d53-4c10-b2ed-3b84e8612684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:41.857906 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 896/1756] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:41 2022] POST /identity/v3/projects => generated 377 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:58:41.859923 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e9a6ac-8e8c-4269-b31f-e979936efd5a admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:41.888945 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f206791-2d53-4c10-b2ed-3b84e8612684 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:41.966383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e9a6ac-8e8c-4269-b31f-e979936efd5a admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:41.975338 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7e8897-8dae-482a-b2a8-fd5e87baf7a2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:41.975952 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7e8897-8dae-482a-b2a8-fd5e87baf7a2 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:41.976226 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7e8897-8dae-482a-b2a8-fd5e87baf7a2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:41.976494 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7e8897-8dae-482a-b2a8-fd5e87baf7a2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:41.982899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa23cc6-b18d-49ea-94e6-61ba5d095e51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:41.984735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa23cc6-b18d-49ea-94e6-61ba5d095e51 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:41.984874 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa23cc6-b18d-49ea-94e6-61ba5d095e51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:41.985116 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa23cc6-b18d-49ea-94e6-61ba5d095e51 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:41.985774 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa23cc6-b18d-49ea-94e6-61ba5d095e51 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:41.988477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa23cc6-b18d-49ea-94e6-61ba5d095e51 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:41.995838 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:41.999133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7e8897-8dae-482a-b2a8-fd5e87baf7a2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:42.000911 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7e8897-8dae-482a-b2a8-fd5e87baf7a2 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:42.014500 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 861/1757] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:42.015394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7e8897-8dae-482a-b2a8-fd5e87baf7a2 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:42.017374 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9b7e8897-8dae-482a-b2a8-fd5e87baf7a2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:42.027616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 897/1758] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:41 2022] POST /identity/v3/users => generated 289 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:42.031478 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa23cc6-b18d-49ea-94e6-61ba5d095e51 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:42.037623 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7e8897-8dae-482a-b2a8-fd5e87baf7a2 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:42.043309 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccf3ca5-abe4-4798-9344-4fda2d0c2622 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:42.043952 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccf3ca5-abe4-4798-9344-4fda2d0c2622 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.044342 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccf3ca5-abe4-4798-9344-4fda2d0c2622 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.044464 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa23cc6-b18d-49ea-94e6-61ba5d095e51 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:42.044629 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccf3ca5-abe4-4798-9344-4fda2d0c2622 admin admin] PATH_INFO: `/v3/projects/217097ae31cf4319b2c373e5fc625d91` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.045798 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccf3ca5-abe4-4798-9344-4fda2d0c2622 admin admin] RBAC: Authorizing `identity:delete_project(project_id=217097ae31cf4319b2c373e5fc625d91)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:42.047904 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccf3ca5-abe4-4798-9344-4fda2d0c2622 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:42.049591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccb3905-4adf-4ff4-bd91-d508c4d48aba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:42.050310 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccb3905-4adf-4ff4-bd91-d508c4d48aba admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.050607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccb3905-4adf-4ff4-bd91-d508c4d48aba admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.050961 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccb3905-4adf-4ff4-bd91-d508c4d48aba admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.051568 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccb3905-4adf-4ff4-bd91-d508c4d48aba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:42.053186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccb3905-4adf-4ff4-bd91-d508c4d48aba admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:42.067098 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 898/1759] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:42 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 00:58:42.068515 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccb3905-4adf-4ff4-bd91-d508c4d48aba admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:42.081612 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccb3905-4adf-4ff4-bd91-d508c4d48aba admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:42.088998 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477d47c2-ee32-46a7-b59b-64ae126543a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:42.089926 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477d47c2-ee32-46a7-b59b-64ae126543a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.090399 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477d47c2-ee32-46a7-b59b-64ae126543a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.090886 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477d47c2-ee32-46a7-b59b-64ae126543a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.091717 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477d47c2-ee32-46a7-b59b-64ae126543a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:42.093524 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477d47c2-ee32-46a7-b59b-64ae126543a9 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:42.104165 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cccf3ca5-abe4-4798-9344-4fda2d0c2622 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '217097ae31cf4319b2c373e5fc625d91', 'request_id': 'req-cccf3ca5-abe4-4798-9344-4fda2d0c2622'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:42.107596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 899/1760] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:42 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 00:58:42.112466 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477d47c2-ee32-46a7-b59b-64ae126543a9 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:42.117244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cccf3ca5-abe4-4798-9344-4fda2d0c2622 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '217097ae31cf4319b2c373e5fc625d91', 'request_id': 'req-cccf3ca5-abe4-4798-9344-4fda2d0c2622'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:42.123796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477d47c2-ee32-46a7-b59b-64ae126543a9 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:42.125883 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cccf3ca5-abe4-4798-9344-4fda2d0c2622 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '217097ae31cf4319b2c373e5fc625d91', 'request_id': 'req-cccf3ca5-abe4-4798-9344-4fda2d0c2622'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:42.136805 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:42.144026 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cccf3ca5-abe4-4798-9344-4fda2d0c2622 admin admin] The token cache is being invalidate because project 217097ae31cf4319b2c373e5fc625d91 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:42.144493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cccf3ca5-abe4-4798-9344-4fda2d0c2622 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:42.145280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cccf3ca5-abe4-4798-9344-4fda2d0c2622 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:42.147273 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 862/1761] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:42 2022] DELETE /identity/v3/projects/217097ae31cf4319b2c373e5fc625d91 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:42.152789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c03484-d953-4bf9-b3ac-193b3a9f8dde None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.153082 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c03484-d953-4bf9-b3ac-193b3a9f8dde None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.153360 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c03484-d953-4bf9-b3ac-193b3a9f8dde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.175472 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-05c03484-d953-4bf9-b3ac-193b3a9f8dde None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:42.176857 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-05c03484-d953-4bf9-b3ac-193b3a9f8dde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:42.213761 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8167eeaf-ae6f-4ea9-9db8-68b2969360c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:42.214637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8167eeaf-ae6f-4ea9-9db8-68b2969360c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.215075 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8167eeaf-ae6f-4ea9-9db8-68b2969360c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.215456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8167eeaf-ae6f-4ea9-9db8-68b2969360c7 admin admin] PATH_INFO: `/v3/projects/103f30a9d97c4082ba59a10cc3bc218b/users/7bb90895b6864fe086f56cf6f4ab6172/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.233506 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8167eeaf-ae6f-4ea9-9db8-68b2969360c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=103f30a9d97c4082ba59a10cc3bc218b, user_id=7bb90895b6864fe086f56cf6f4ab6172, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:42.236524 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8167eeaf-ae6f-4ea9-9db8-68b2969360c7 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:42.264212 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 900/1762] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:42 2022] PUT /identity/v3/projects/103f30a9d97c4082ba59a10cc3bc218b/users/7bb90895b6864fe086f56cf6f4ab6172/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:42.265151 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 863/1763] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:42.273886 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de681780-1948-43b8-8b73-c48262886f14 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.274316 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de681780-1948-43b8-8b73-c48262886f14 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.274598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de681780-1948-43b8-8b73-c48262886f14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.277745 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1928d7-5075-4d98-b0a8-e51c29c8425d None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.277745 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1928d7-5075-4d98-b0a8-e51c29c8425d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.277745 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1928d7-5075-4d98-b0a8-e51c29c8425d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.304640 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-de681780-1948-43b8-8b73-c48262886f14 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:42.306398 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-de681780-1948-43b8-8b73-c48262886f14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:42.313228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-dc1928d7-5075-4d98-b0a8-e51c29c8425d None None] MFA Rules not processed for user `7bb90895b6864fe086f56cf6f4ab6172`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:42.315189 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-dc1928d7-5075-4d98-b0a8-e51c29c8425d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:42.414442 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 901/1764] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:42.418475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de681780-1948-43b8-8b73-c48262886f14 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:42.430964 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de681780-1948-43b8-8b73-c48262886f14 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:42.437099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af6fe4d-1f4a-4cc6-8061-c357e5a78b22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:42.437734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af6fe4d-1f4a-4cc6-8061-c357e5a78b22 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.438354 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af6fe4d-1f4a-4cc6-8061-c357e5a78b22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.438354 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af6fe4d-1f4a-4cc6-8061-c357e5a78b22 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.439318 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af6fe4d-1f4a-4cc6-8061-c357e5a78b22 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:58:42.439559 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af6fe4d-1f4a-4cc6-8061-c357e5a78b22 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:42.441610 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af6fe4d-1f4a-4cc6-8061-c357e5a78b22 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:42.448295 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 864/1765] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:42 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:42.452779 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1928d7-5075-4d98-b0a8-e51c29c8425d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:42.455201 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-dc1928d7-5075-4d98-b0a8-e51c29c8425d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:42.459402 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 902/1766] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:58:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:58:42.461161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af6fe4d-1f4a-4cc6-8061-c357e5a78b22 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:42.474198 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af6fe4d-1f4a-4cc6-8061-c357e5a78b22 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:42.480595 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2bdbca-b444-4c8d-9604-370572fcb550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:42.481054 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2bdbca-b444-4c8d-9604-370572fcb550 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.483287 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1928d7-5075-4d98-b0a8-e51c29c8425d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:42.484929 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2bdbca-b444-4c8d-9604-370572fcb550 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.485522 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2bdbca-b444-4c8d-9604-370572fcb550 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.491315 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a951108-5c2f-437a-8c34-53d4d4e8c645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:42.492013 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a951108-5c2f-437a-8c34-53d4d4e8c645 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.492322 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a951108-5c2f-437a-8c34-53d4d4e8c645 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.492599 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a951108-5c2f-437a-8c34-53d4d4e8c645 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.493398 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a951108-5c2f-437a-8c34-53d4d4e8c645 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:42.495180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2bdbca-b444-4c8d-9604-370572fcb550 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:42.495993 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a951108-5c2f-437a-8c34-53d4d4e8c645 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:42.497993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2bdbca-b444-4c8d-9604-370572fcb550 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:42.504673 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:42.536912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 865/1767] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:42 2022] POST /identity/v3/projects => generated 377 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:42.536912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a951108-5c2f-437a-8c34-53d4d4e8c645 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:42.537505 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0a951108-5c2f-437a-8c34-53d4d4e8c645 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:42.547519 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 903/1768] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:42 2022] POST /identity/v3/projects => generated 367 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:42.548424 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2bdbca-b444-4c8d-9604-370572fcb550 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:42.633791 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a951108-5c2f-437a-8c34-53d4d4e8c645 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:42.671135 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2bdbca-b444-4c8d-9604-370572fcb550 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:42.677077 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae33a45-313f-4742-8fde-28d6afa980d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:42.677760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae33a45-313f-4742-8fde-28d6afa980d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.678129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae33a45-313f-4742-8fde-28d6afa980d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.678629 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae33a45-313f-4742-8fde-28d6afa980d3 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.679427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae33a45-313f-4742-8fde-28d6afa980d3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:42.681503 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae33a45-313f-4742-8fde-28d6afa980d3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:42.693898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:42.703990 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196763a1-50b2-48dc-b913-eec6d34079b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:42.704810 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196763a1-50b2-48dc-b913-eec6d34079b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.705097 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196763a1-50b2-48dc-b913-eec6d34079b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.705408 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196763a1-50b2-48dc-b913-eec6d34079b4 admin admin] PATH_INFO: `/v3/projects/809f1bfaf9db45ccaa61dd5d5aa0cee7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.706604 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196763a1-50b2-48dc-b913-eec6d34079b4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=809f1bfaf9db45ccaa61dd5d5aa0cee7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:42.708771 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196763a1-50b2-48dc-b913-eec6d34079b4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:42.710942 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 904/1769] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:42 2022] POST /identity/v3/users => generated 289 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:42.712295 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae33a45-313f-4742-8fde-28d6afa980d3 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:42.724719 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae33a45-313f-4742-8fde-28d6afa980d3 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:42.730350 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992e446b-59c5-4fdc-b020-2de43ff0209e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:42.731035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992e446b-59c5-4fdc-b020-2de43ff0209e admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.731321 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992e446b-59c5-4fdc-b020-2de43ff0209e admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.731593 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992e446b-59c5-4fdc-b020-2de43ff0209e admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.732242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992e446b-59c5-4fdc-b020-2de43ff0209e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:42.734208 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992e446b-59c5-4fdc-b020-2de43ff0209e admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:42.761005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 905/1770] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:42 2022] POST /identity/v3/users => generated 284 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:42.762308 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992e446b-59c5-4fdc-b020-2de43ff0209e admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:42.773044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992e446b-59c5-4fdc-b020-2de43ff0209e admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:42.779389 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-196763a1-50b2-48dc-b913-eec6d34079b4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '809f1bfaf9db45ccaa61dd5d5aa0cee7', 'request_id': 'req-196763a1-50b2-48dc-b913-eec6d34079b4'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:42.788610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-196763a1-50b2-48dc-b913-eec6d34079b4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '809f1bfaf9db45ccaa61dd5d5aa0cee7', 'request_id': 'req-196763a1-50b2-48dc-b913-eec6d34079b4'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:42.799138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-196763a1-50b2-48dc-b913-eec6d34079b4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '809f1bfaf9db45ccaa61dd5d5aa0cee7', 'request_id': 'req-196763a1-50b2-48dc-b913-eec6d34079b4'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:42.816086 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-196763a1-50b2-48dc-b913-eec6d34079b4 admin admin] The token cache is being invalidate because project 809f1bfaf9db45ccaa61dd5d5aa0cee7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:42.816289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-196763a1-50b2-48dc-b913-eec6d34079b4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:42.817808 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-196763a1-50b2-48dc-b913-eec6d34079b4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:42.819836 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 866/1771] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:42 2022] DELETE /identity/v3/projects/809f1bfaf9db45ccaa61dd5d5aa0cee7 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:42.820639 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196763a1-50b2-48dc-b913-eec6d34079b4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:42.821993 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-196763a1-50b2-48dc-b913-eec6d34079b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:42.855528 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:42.872972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7279d9-158f-4016-8565-fdd855585af5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:42.873748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7279d9-158f-4016-8565-fdd855585af5 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.874097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7279d9-158f-4016-8565-fdd855585af5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.874437 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7279d9-158f-4016-8565-fdd855585af5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.875132 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7279d9-158f-4016-8565-fdd855585af5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:42.876854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7279d9-158f-4016-8565-fdd855585af5 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:42.889703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 906/1772] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:42 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:42.890589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7279d9-158f-4016-8565-fdd855585af5 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:42.895392 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-5d7279d9-158f-4016-8565-fdd855585af5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:42.909644 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7279d9-158f-4016-8565-fdd855585af5 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:42.910004 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196763a1-50b2-48dc-b913-eec6d34079b4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:42.914289 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0ac7ef-dc0a-4b3a-a4b2-7c367a80e327 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:42.914699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0250c963-8a91-4c79-bf20-343f14e330a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:42.914955 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0ac7ef-dc0a-4b3a-a4b2-7c367a80e327 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.915246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0ac7ef-dc0a-4b3a-a4b2-7c367a80e327 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.915382 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0250c963-8a91-4c79-bf20-343f14e330a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.915523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0ac7ef-dc0a-4b3a-a4b2-7c367a80e327 admin admin] PATH_INFO: `/v3/users/43fe577eb78a41aeb1b53d723c4e523e` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.915632 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0250c963-8a91-4c79-bf20-343f14e330a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.915874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0250c963-8a91-4c79-bf20-343f14e330a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.916667 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0250c963-8a91-4c79-bf20-343f14e330a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:42.916941 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0ac7ef-dc0a-4b3a-a4b2-7c367a80e327 admin admin] RBAC: Authorizing `identity:delete_user(user_id=43fe577eb78a41aeb1b53d723c4e523e)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:42.918270 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0250c963-8a91-4c79-bf20-343f14e330a2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:42.919143 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0ac7ef-dc0a-4b3a-a4b2-7c367a80e327 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:42.928871 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 867/1773] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:42 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:42.929730 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0250c963-8a91-4c79-bf20-343f14e330a2 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:42.931577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0250c963-8a91-4c79-bf20-343f14e330a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:42.945907 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0250c963-8a91-4c79-bf20-343f14e330a2 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:42.952331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc80711-cd70-4cf1-a85a-517b81251f9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:42.952943 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc80711-cd70-4cf1-a85a-517b81251f9e admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.953224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc80711-cd70-4cf1-a85a-517b81251f9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.953495 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc80711-cd70-4cf1-a85a-517b81251f9e admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.954131 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc80711-cd70-4cf1-a85a-517b81251f9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:42.955863 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc80711-cd70-4cf1-a85a-517b81251f9e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:42.966410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 868/1774] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:42 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:42.967344 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc80711-cd70-4cf1-a85a-517b81251f9e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:42.975626 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3f0ac7ef-dc0a-4b3a-a4b2-7c367a80e327 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '43fe577eb78a41aeb1b53d723c4e523e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:42.976308 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3f0ac7ef-dc0a-4b3a-a4b2-7c367a80e327 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '43fe577eb78a41aeb1b53d723c4e523e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:42.976908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc80711-cd70-4cf1-a85a-517b81251f9e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:42.983195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6c0c09-8fed-49ca-9f21-37b1a5d040bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:42.983837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6c0c09-8fed-49ca-9f21-37b1a5d040bc admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:42.984129 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6c0c09-8fed-49ca-9f21-37b1a5d040bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:42.984400 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6c0c09-8fed-49ca-9f21-37b1a5d040bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:42.985044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6c0c09-8fed-49ca-9f21-37b1a5d040bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:42.985156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3f0ac7ef-dc0a-4b3a-a4b2-7c367a80e327 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '43fe577eb78a41aeb1b53d723c4e523e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:42.986771 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6c0c09-8fed-49ca-9f21-37b1a5d040bc admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:42.998692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3f0ac7ef-dc0a-4b3a-a4b2-7c367a80e327 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '43fe577eb78a41aeb1b53d723c4e523e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:42.998889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 869/1775] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:42 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:42.999639 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6c0c09-8fed-49ca-9f21-37b1a5d040bc admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:43.006446 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3f0ac7ef-dc0a-4b3a-a4b2-7c367a80e327 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '43fe577eb78a41aeb1b53d723c4e523e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:43.010324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6c0c09-8fed-49ca-9f21-37b1a5d040bc admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:43.013299 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 907/1776] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:42 2022] DELETE /identity/v3/users/43fe577eb78a41aeb1b53d723c4e523e => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:43.014034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0ac7ef-dc0a-4b3a-a4b2-7c367a80e327 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:43.015466 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-3f0ac7ef-dc0a-4b3a-a4b2-7c367a80e327 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:43.017968 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3f1e61-d8eb-4926-8b59-9fee68d3bd2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:43.018675 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3f1e61-d8eb-4926-8b59-9fee68d3bd2d admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:43.018984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3f1e61-d8eb-4926-8b59-9fee68d3bd2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:43.019260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3f1e61-d8eb-4926-8b59-9fee68d3bd2d admin admin] PATH_INFO: `/v3/projects/c4cc37c4e35e4247838082ded1a71a2c/users/54f34cb777e648d892a848a44b6df3aa/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:43.037028 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3f1e61-d8eb-4926-8b59-9fee68d3bd2d admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4cc37c4e35e4247838082ded1a71a2c, user_id=54f34cb777e648d892a848a44b6df3aa, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:43.039846 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3f1e61-d8eb-4926-8b59-9fee68d3bd2d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:43.055690 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 870/1777] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:42 2022] PUT /identity/v3/projects/c4cc37c4e35e4247838082ded1a71a2c/users/54f34cb777e648d892a848a44b6df3aa/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:43.068172 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5deecd49-3e6b-4743-962b-011cf3f9a4bf None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:43.068784 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5deecd49-3e6b-4743-962b-011cf3f9a4bf None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:43.068784 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5deecd49-3e6b-4743-962b-011cf3f9a4bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:43.078963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:INSERT=6,keystone:SELECT=84,keystone:DELETE=10 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:43.102589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0ac7ef-dc0a-4b3a-a4b2-7c367a80e327 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:43.104995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5deecd49-3e6b-4743-962b-011cf3f9a4bf None None] MFA Rules not processed for user `54f34cb777e648d892a848a44b6df3aa`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:43.106424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5deecd49-3e6b-4743-962b-011cf3f9a4bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:43.186671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cf369a-ab22-4a2f-bdd6-3e95764c4133 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:43.187411 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cf369a-ab22-4a2f-bdd6-3e95764c4133 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:43.187692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cf369a-ab22-4a2f-bdd6-3e95764c4133 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:43.187964 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cf369a-ab22-4a2f-bdd6-3e95764c4133 admin admin] PATH_INFO: `/v3/projects/c1d5006e67c24fdfb26bcfec851701d2/users/aeada197e2224f8b936aa8ff2176a54e/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:43.203730 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 871/1778] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:43 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:58:43.206654 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cf369a-ab22-4a2f-bdd6-3e95764c4133 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1d5006e67c24fdfb26bcfec851701d2, user_id=aeada197e2224f8b936aa8ff2176a54e, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:43.208318 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5deecd49-3e6b-4743-962b-011cf3f9a4bf None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:43.209692 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5deecd49-3e6b-4743-962b-011cf3f9a4bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:43.209891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cf369a-ab22-4a2f-bdd6-3e95764c4133 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:43.227729 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5deecd49-3e6b-4743-962b-011cf3f9a4bf None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:43.229424 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 908/1779] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:43 2022] PUT /identity/v3/projects/c1d5006e67c24fdfb26bcfec851701d2/users/aeada197e2224f8b936aa8ff2176a54e/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:43.245527 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af0338b-b7c3-4825-b065-1e2121c395f4 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:43.245953 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af0338b-b7c3-4825-b065-1e2121c395f4 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:43.246235 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af0338b-b7c3-4825-b065-1e2121c395f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:43.282257 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-1af0338b-b7c3-4825-b065-1e2121c395f4 None None] MFA Rules not processed for user `aeada197e2224f8b936aa8ff2176a54e`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:43.283995 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-1af0338b-b7c3-4825-b065-1e2121c395f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:43.334038 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2853f6bb-6d37-4575-ab39-1327e5f13a6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:43.334713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2853f6bb-6d37-4575-ab39-1327e5f13a6d admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:43.335041 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2853f6bb-6d37-4575-ab39-1327e5f13a6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:43.335384 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2853f6bb-6d37-4575-ab39-1327e5f13a6d admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:43.344451 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2853f6bb-6d37-4575-ab39-1327e5f13a6d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:43.346380 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2853f6bb-6d37-4575-ab39-1327e5f13a6d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:43.358266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:43.387083 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 872/1780] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:43 2022] POST /identity/v3/projects => generated 379 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:43.388549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2853f6bb-6d37-4575-ab39-1327e5f13a6d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:43.390792 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2853f6bb-6d37-4575-ab39-1327e5f13a6d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:43.392742 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 909/1781] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:43 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:58:43.396673 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af0338b-b7c3-4825-b065-1e2121c395f4 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:43.418505 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af0338b-b7c3-4825-b065-1e2121c395f4 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:43.462564 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2853f6bb-6d37-4575-ab39-1327e5f13a6d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:43.468311 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b0f93b-6f64-4c22-a34f-d363da0212a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:43.468957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b0f93b-6f64-4c22-a34f-d363da0212a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:43.469239 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b0f93b-6f64-4c22-a34f-d363da0212a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:43.469513 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b0f93b-6f64-4c22-a34f-d363da0212a8 admin admin] PATH_INFO: `/v3/projects/d4191f6b320a49b38e96260733c96dfd` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:43.470690 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b0f93b-6f64-4c22-a34f-d363da0212a8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d4191f6b320a49b38e96260733c96dfd)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:43.472653 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b0f93b-6f64-4c22-a34f-d363da0212a8 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:43.484301 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9777bea0-0a18-4d88-b679-6477a6a9658f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:43.484927 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9777bea0-0a18-4d88-b679-6477a6a9658f admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:43.485216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9777bea0-0a18-4d88-b679-6477a6a9658f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:43.485491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9777bea0-0a18-4d88-b679-6477a6a9658f admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:43.486119 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9777bea0-0a18-4d88-b679-6477a6a9658f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:43.488111 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9777bea0-0a18-4d88-b679-6477a6a9658f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:43.497621 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:43.520847 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 910/1782] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:43 2022] POST /identity/v3/users => generated 296 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:43.522027 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9777bea0-0a18-4d88-b679-6477a6a9658f admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:43.524716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-9777bea0-0a18-4d88-b679-6477a6a9658f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:43.537750 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-76b0f93b-6f64-4c22-a34f-d363da0212a8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd4191f6b320a49b38e96260733c96dfd', 'request_id': 'req-76b0f93b-6f64-4c22-a34f-d363da0212a8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:43.552848 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-76b0f93b-6f64-4c22-a34f-d363da0212a8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd4191f6b320a49b38e96260733c96dfd', 'request_id': 'req-76b0f93b-6f64-4c22-a34f-d363da0212a8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:43.560936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-76b0f93b-6f64-4c22-a34f-d363da0212a8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd4191f6b320a49b38e96260733c96dfd', 'request_id': 'req-76b0f93b-6f64-4c22-a34f-d363da0212a8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:43.573151 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-76b0f93b-6f64-4c22-a34f-d363da0212a8 admin admin] The token cache is being invalidate because project d4191f6b320a49b38e96260733c96dfd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:43.573716 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-76b0f93b-6f64-4c22-a34f-d363da0212a8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:43.574476 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-76b0f93b-6f64-4c22-a34f-d363da0212a8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:43.576441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 873/1783] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:43 2022] DELETE /identity/v3/projects/d4191f6b320a49b38e96260733c96dfd => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:43.577215 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b0f93b-6f64-4c22-a34f-d363da0212a8 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:43.578594 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-76b0f93b-6f64-4c22-a34f-d363da0212a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:43.609777 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9777bea0-0a18-4d88-b679-6477a6a9658f admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:43.648030 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b0f93b-6f64-4c22-a34f-d363da0212a8 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:43.655554 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce6b2f0-7f1c-4092-9aca-103df07f4ec5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:43.656321 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce6b2f0-7f1c-4092-9aca-103df07f4ec5 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:43.656605 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce6b2f0-7f1c-4092-9aca-103df07f4ec5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:43.656879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce6b2f0-7f1c-4092-9aca-103df07f4ec5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:43.657539 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce6b2f0-7f1c-4092-9aca-103df07f4ec5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:43.659448 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce6b2f0-7f1c-4092-9aca-103df07f4ec5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:43.669834 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98720d2f-1639-412b-ae4d-25261c486a23 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '78473c29cef449b89bc9185de67f1bf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:43.670717 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98720d2f-1639-412b-ae4d-25261c486a23 service glance] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:43.671110 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98720d2f-1639-412b-ae4d-25261c486a23 service glance] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:43.671453 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98720d2f-1639-412b-ae4d-25261c486a23 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:43.673561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-98720d2f-1639-412b-ae4d-25261c486a23 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:43.677022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 874/1784] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:43 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:43.677022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8d5072-1c29-42e8-96c0-47ac76e6c8ba None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:43.677022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8d5072-1c29-42e8-96c0-47ac76e6c8ba None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:43.677022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8d5072-1c29-42e8-96c0-47ac76e6c8ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:43.726560 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-1e8d5072-1c29-42e8-96c0-47ac76e6c8ba None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:43.727775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1e8d5072-1c29-42e8-96c0-47ac76e6c8ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:43.736823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 875/1785] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:43.737665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8d5072-1c29-42e8-96c0-47ac76e6c8ba None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:43.748028 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8d5072-1c29-42e8-96c0-47ac76e6c8ba None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:43.754151 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c8a70c-572b-43e2-915f-cf0682637c9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:43.754819 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c8a70c-572b-43e2-915f-cf0682637c9f admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:43.755130 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c8a70c-572b-43e2-915f-cf0682637c9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:43.755409 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c8a70c-572b-43e2-915f-cf0682637c9f admin admin] PATH_INFO: `/v3/projects/0256e137c4ef4828823b77ceb13e5504/users/18ad8f619c6a4955870413995be857ba/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:43.762479 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:43.765235 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98720d2f-1639-412b-ae4d-25261c486a23 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:43.766791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98720d2f-1639-412b-ae4d-25261c486a23 service glance] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:43.777798 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 911/1786] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:43 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:43.779203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffebff19-0fa2-42d2-8a41-99909fcb271c None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:43.779601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffebff19-0fa2-42d2-8a41-99909fcb271c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:43.779984 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffebff19-0fa2-42d2-8a41-99909fcb271c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:43.782995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c8a70c-572b-43e2-915f-cf0682637c9f admin admin] RBAC: Authorizing `identity:create_grant(project_id=0256e137c4ef4828823b77ceb13e5504, user_id=18ad8f619c6a4955870413995be857ba, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:43.786222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c8a70c-572b-43e2-915f-cf0682637c9f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:43.805651 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 876/1787] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:43 2022] PUT /identity/v3/projects/0256e137c4ef4828823b77ceb13e5504/users/18ad8f619c6a4955870413995be857ba/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:43.810820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c8a70c-572b-43e2-915f-cf0682637c9f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:43.812267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-ffebff19-0fa2-42d2-8a41-99909fcb271c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:43.815086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-ffebff19-0fa2-42d2-8a41-99909fcb271c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:43.897500 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 912/1788] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:43.899656 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffebff19-0fa2-42d2-8a41-99909fcb271c None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:43.907395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:43.915942 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffebff19-0fa2-42d2-8a41-99909fcb271c None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:43.919106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c8a70c-572b-43e2-915f-cf0682637c9f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:43.923224 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292c4c42-826c-4c01-9e66-d625e04f72f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:43.924380 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292c4c42-826c-4c01-9e66-d625e04f72f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:43.924925 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292c4c42-826c-4c01-9e66-d625e04f72f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:43.925313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292c4c42-826c-4c01-9e66-d625e04f72f7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:43.926065 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292c4c42-826c-4c01-9e66-d625e04f72f7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:58:43.926552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292c4c42-826c-4c01-9e66-d625e04f72f7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:43.927460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b76cb0b-917a-4654-a146-559fc0afc817 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:43.928307 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b76cb0b-917a-4654-a146-559fc0afc817 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:43.928798 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b76cb0b-917a-4654-a146-559fc0afc817 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:43.929025 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292c4c42-826c-4c01-9e66-d625e04f72f7 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:43.929463 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b76cb0b-917a-4654-a146-559fc0afc817 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:43.931283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b76cb0b-917a-4654-a146-559fc0afc817 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:43.934949 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b76cb0b-917a-4654-a146-559fc0afc817 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:43.952332 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 877/1789] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:43 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:43.956570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b76cb0b-917a-4654-a146-559fc0afc817 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:43.962035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 913/1790] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:58:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:58:43.967526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292c4c42-826c-4c01-9e66-d625e04f72f7 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:43.981598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292c4c42-826c-4c01-9e66-d625e04f72f7 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:43.983703 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b76cb0b-917a-4654-a146-559fc0afc817 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:43.988522 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41443e46-d673-4aaa-b398-687e1fd4df6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:43.988522 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41443e46-d673-4aaa-b398-687e1fd4df6d admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:43.988828 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41443e46-d673-4aaa-b398-687e1fd4df6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:43.989042 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41443e46-d673-4aaa-b398-687e1fd4df6d admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:43.989676 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41443e46-d673-4aaa-b398-687e1fd4df6d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:43.991723 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41443e46-d673-4aaa-b398-687e1fd4df6d admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:43.994082 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6447884-78a3-4a41-9edf-d900db063108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:43.995462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6447884-78a3-4a41-9edf-d900db063108 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:43.996035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6447884-78a3-4a41-9edf-d900db063108 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:43.996677 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6447884-78a3-4a41-9edf-d900db063108 admin admin] PATH_INFO: `/v3/projects/0256e137c4ef4828823b77ceb13e5504/users/18ad8f619c6a4955870413995be857ba/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:43.999731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6447884-78a3-4a41-9edf-d900db063108 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0256e137c4ef4828823b77ceb13e5504, user_id=18ad8f619c6a4955870413995be857ba, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:44.003257 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6447884-78a3-4a41-9edf-d900db063108 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:44.019469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 878/1791] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:43 2022] PUT /identity/v3/projects/0256e137c4ef4828823b77ceb13e5504/users/18ad8f619c6a4955870413995be857ba/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:58:44.030372 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f403ece-ffae-479b-93d5-c152120e4f66 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:44.030921 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 914/1792] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:43 2022] POST /identity/v3/projects => generated 363 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:44.033804 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f403ece-ffae-479b-93d5-c152120e4f66 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:44.034199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f403ece-ffae-479b-93d5-c152120e4f66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:44.035512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41443e46-d673-4aaa-b398-687e1fd4df6d admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:44.071869 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-6f403ece-ffae-479b-93d5-c152120e4f66 None None] MFA Rules not processed for user `18ad8f619c6a4955870413995be857ba`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:44.073811 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6f403ece-ffae-479b-93d5-c152120e4f66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:44.133333 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41443e46-d673-4aaa-b398-687e1fd4df6d admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:44.139011 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e58004-8b01-46b4-a06d-a8f824e7e6da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:44.139801 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e58004-8b01-46b4-a06d-a8f824e7e6da admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:44.140157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e58004-8b01-46b4-a06d-a8f824e7e6da admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:44.140503 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e58004-8b01-46b4-a06d-a8f824e7e6da admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:44.141249 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e58004-8b01-46b4-a06d-a8f824e7e6da admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:44.143690 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e58004-8b01-46b4-a06d-a8f824e7e6da admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:44.153514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 879/1793] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:44 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:58:44.164957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f403ece-ffae-479b-93d5-c152120e4f66 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:44.166635 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6f403ece-ffae-479b-93d5-c152120e4f66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:44.169582 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 915/1794] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:44 2022] POST /identity/v3/users => generated 282 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:58:44.173759 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e58004-8b01-46b4-a06d-a8f824e7e6da admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:44.185691 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e58004-8b01-46b4-a06d-a8f824e7e6da admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:44.193183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dacdf3f-c51a-4ef0-86a4-dde42bd1ce32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:44.194113 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dacdf3f-c51a-4ef0-86a4-dde42bd1ce32 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:44.194650 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dacdf3f-c51a-4ef0-86a4-dde42bd1ce32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:44.195079 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dacdf3f-c51a-4ef0-86a4-dde42bd1ce32 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:44.195955 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dacdf3f-c51a-4ef0-86a4-dde42bd1ce32 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:44.198734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dacdf3f-c51a-4ef0-86a4-dde42bd1ce32 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:44.210855 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 916/1795] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:44 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 00:58:44.214988 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dacdf3f-c51a-4ef0-86a4-dde42bd1ce32 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:44.226933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dacdf3f-c51a-4ef0-86a4-dde42bd1ce32 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:44.234508 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0642e484-a9fa-4e8c-9f40-ef559dd926c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:44.235280 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0642e484-a9fa-4e8c-9f40-ef559dd926c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:44.235280 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0642e484-a9fa-4e8c-9f40-ef559dd926c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:44.236014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0642e484-a9fa-4e8c-9f40-ef559dd926c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:44.236335 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0642e484-a9fa-4e8c-9f40-ef559dd926c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:44.239025 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0642e484-a9fa-4e8c-9f40-ef559dd926c7 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:44.249084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f403ece-ffae-479b-93d5-c152120e4f66 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:44.252974 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 917/1796] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:44 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:44.254964 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32669b9a-1a52-4adf-ab31-eb787a4c3e38 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:44.255628 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32669b9a-1a52-4adf-ab31-eb787a4c3e38 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:44.255907 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32669b9a-1a52-4adf-ab31-eb787a4c3e38 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:44.256454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32669b9a-1a52-4adf-ab31-eb787a4c3e38 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:44.257232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0642e484-a9fa-4e8c-9f40-ef559dd926c7 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:44.266284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:44.269151 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0642e484-a9fa-4e8c-9f40-ef559dd926c7 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:44.269579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32669b9a-1a52-4adf-ab31-eb787a4c3e38 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:44.272316 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32669b9a-1a52-4adf-ab31-eb787a4c3e38 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:44.275567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbd0022-38ac-423c-ac2f-c42df76f162a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:44.276628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbd0022-38ac-423c-ac2f-c42df76f162a admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:44.276984 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbd0022-38ac-423c-ac2f-c42df76f162a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:44.277300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbd0022-38ac-423c-ac2f-c42df76f162a admin admin] PATH_INFO: `/v3/projects/411d2dda80534b15946c38b816defaa1/users/ff4a9c6cf0a54f4a98ef180ae6402ab3/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:44.283925 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 880/1797] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:44 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:58:44.293718 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbd0022-38ac-423c-ac2f-c42df76f162a admin admin] RBAC: Authorizing `identity:create_grant(project_id=411d2dda80534b15946c38b816defaa1, user_id=ff4a9c6cf0a54f4a98ef180ae6402ab3, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:44.296731 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbd0022-38ac-423c-ac2f-c42df76f162a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:44.322817 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 918/1798] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:44 2022] PUT /identity/v3/projects/411d2dda80534b15946c38b816defaa1/users/ff4a9c6cf0a54f4a98ef180ae6402ab3/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:58:44.331727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acfe2aa-e0a1-489d-a130-a9ac0b303c20 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:44.332013 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acfe2aa-e0a1-489d-a130-a9ac0b303c20 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:44.332286 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acfe2aa-e0a1-489d-a130-a9ac0b303c20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:44.354133 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbd0022-38ac-423c-ac2f-c42df76f162a admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:44.376208 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4acfe2aa-e0a1-489d-a130-a9ac0b303c20 None None] MFA Rules not processed for user `ff4a9c6cf0a54f4a98ef180ae6402ab3`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:44.377858 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4acfe2aa-e0a1-489d-a130-a9ac0b303c20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:44.441796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbd0022-38ac-423c-ac2f-c42df76f162a admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:44.447058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4015283-3d80-4dfe-9f06-c8dbc389116a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:44.448924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4015283-3d80-4dfe-9f06-c8dbc389116a service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:44.449588 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4015283-3d80-4dfe-9f06-c8dbc389116a service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:44.450081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4015283-3d80-4dfe-9f06-c8dbc389116a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:44.452358 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d4015283-3d80-4dfe-9f06-c8dbc389116a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:44.490906 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:44.495220 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 881/1799] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:44 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:44.504563 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acfe2aa-e0a1-489d-a130-a9ac0b303c20 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:44.506024 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4acfe2aa-e0a1-489d-a130-a9ac0b303c20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:44.519238 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acfe2aa-e0a1-489d-a130-a9ac0b303c20 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:44.524605 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9c8208-6ae0-4caf-b293-6a57d053b0a7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:44.525662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9c8208-6ae0-4caf-b293-6a57d053b0a7 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:44.525988 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9c8208-6ae0-4caf-b293-6a57d053b0a7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:44.526345 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9c8208-6ae0-4caf-b293-6a57d053b0a7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:44.536141 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9c8208-6ae0-4caf-b293-6a57d053b0a7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:44.537673 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9c8208-6ae0-4caf-b293-6a57d053b0a7 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:44.542398 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4015283-3d80-4dfe-9f06-c8dbc389116a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:44.543886 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4015283-3d80-4dfe-9f06-c8dbc389116a service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:44.547958 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 882/1800] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:44 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:58:44.553678 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 919/1801] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:44 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:58:46.320230 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c00267d-29ab-45b1-81db-6148f07b7bf0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:46.321368 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c00267d-29ab-45b1-81db-6148f07b7bf0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:46.321851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c00267d-29ab-45b1-81db-6148f07b7bf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:46.346500 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5c00267d-29ab-45b1-81db-6148f07b7bf0 None None] MFA Rules not processed for user `33b136afbf8a4d98a9d62053b68707e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:46.347775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5c00267d-29ab-45b1-81db-6148f07b7bf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:46.439104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 883/1802] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 16 00:58:46 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:46.494831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4015283-3d80-4dfe-9f06-c8dbc389116a service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:46.497089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d4015283-3d80-4dfe-9f06-c8dbc389116a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:46.629999 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4015283-3d80-4dfe-9f06-c8dbc389116a service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:46.637342 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffdf759-7c9f-4481-807d-1d69f0eee0de service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de64d498da78448394b0d4d1851a2789', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:46.637975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffdf759-7c9f-4481-807d-1d69f0eee0de service swift] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:46.638260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffdf759-7c9f-4481-807d-1d69f0eee0de service swift] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:46.638678 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffdf759-7c9f-4481-807d-1d69f0eee0de service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:46.653725 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffdf759-7c9f-4481-807d-1d69f0eee0de service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:46.655406 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffdf759-7c9f-4481-807d-1d69f0eee0de service swift] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:46.669546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 920/1803] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 16 00:58:46 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 16 00:58:47.288280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d83a72-eed0-4f68-82be-f43d23074ab8 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:47.288569 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d83a72-eed0-4f68-82be-f43d23074ab8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:47.288877 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d83a72-eed0-4f68-82be-f43d23074ab8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:47.313010 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d2d83a72-eed0-4f68-82be-f43d23074ab8 None None] MFA Rules not processed for user `33b136afbf8a4d98a9d62053b68707e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:47.314228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d2d83a72-eed0-4f68-82be-f43d23074ab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:47.322080 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 884/1804] 10.222.0.25 () {58 vars in 967 bytes} [Sun Oct 16 00:58:47 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:47.336140 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffdf759-7c9f-4481-807d-1d69f0eee0de service swift] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:47.351064 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffdf759-7c9f-4481-807d-1d69f0eee0de service swift] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:47.353063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815f4d50-5650-4a86-892d-7f4ced40af1a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de64d498da78448394b0d4d1851a2789', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:47.353706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815f4d50-5650-4a86-892d-7f4ced40af1a service swift] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:47.353986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815f4d50-5650-4a86-892d-7f4ced40af1a service swift] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:47.354260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815f4d50-5650-4a86-892d-7f4ced40af1a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:47.364777 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815f4d50-5650-4a86-892d-7f4ced40af1a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:47.366617 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815f4d50-5650-4a86-892d-7f4ced40af1a service swift] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:47.376735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 921/1805] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 16 00:58:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 16 00:58:47.673399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ca8d3d-f385-4878-b3f9-306299ebe442 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:47.674243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ca8d3d-f385-4878-b3f9-306299ebe442 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:47.674544 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ca8d3d-f385-4878-b3f9-306299ebe442 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:47.695883 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c1ca8d3d-f385-4878-b3f9-306299ebe442 None None] MFA Rules not processed for user `33b136afbf8a4d98a9d62053b68707e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:47.697099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c1ca8d3d-f385-4878-b3f9-306299ebe442 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:47.707362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 885/1806] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 16 00:58:47 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:58:47.726361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815f4d50-5650-4a86-892d-7f4ced40af1a service swift] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:47.737702 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815f4d50-5650-4a86-892d-7f4ced40af1a service swift] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:47.743822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c0dfea-ae2d-4e5d-bf33-858dd0345f2d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de64d498da78448394b0d4d1851a2789', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:47.745540 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c0dfea-ae2d-4e5d-bf33-858dd0345f2d service swift] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:47.745992 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c0dfea-ae2d-4e5d-bf33-858dd0345f2d service swift] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:47.746517 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c0dfea-ae2d-4e5d-bf33-858dd0345f2d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:47.762437 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c0dfea-ae2d-4e5d-bf33-858dd0345f2d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:47.768084 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c0dfea-ae2d-4e5d-bf33-858dd0345f2d service swift] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:47.801646 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 922/1807] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 16 00:58:47 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 16 00:58:47.898323 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ca8d3d-f385-4878-b3f9-306299ebe442 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:47.985415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ca8d3d-f385-4878-b3f9-306299ebe442 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:47.990938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7076c2e-b3f5-47ae-afc6-df36c1a7b4c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:47.991720 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7076c2e-b3f5-47ae-afc6-df36c1a7b4c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:47.992066 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7076c2e-b3f5-47ae-afc6-df36c1a7b4c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:47.992431 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7076c2e-b3f5-47ae-afc6-df36c1a7b4c4 admin admin] PATH_INFO: `/v3/users/7bb90895b6864fe086f56cf6f4ab6172` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:47.993671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7076c2e-b3f5-47ae-afc6-df36c1a7b4c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7bb90895b6864fe086f56cf6f4ab6172)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:47.996635 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7076c2e-b3f5-47ae-afc6-df36c1a7b4c4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:48.003764 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:48.065339 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b7076c2e-b3f5-47ae-afc6-df36c1a7b4c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7bb90895b6864fe086f56cf6f4ab6172'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:48.066305 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b7076c2e-b3f5-47ae-afc6-df36c1a7b4c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7bb90895b6864fe086f56cf6f4ab6172'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:48.075495 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b7076c2e-b3f5-47ae-afc6-df36c1a7b4c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7bb90895b6864fe086f56cf6f4ab6172'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:48.076530 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b7076c2e-b3f5-47ae-afc6-df36c1a7b4c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7bb90895b6864fe086f56cf6f4ab6172'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:48.090543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b7076c2e-b3f5-47ae-afc6-df36c1a7b4c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7bb90895b6864fe086f56cf6f4ab6172'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:48.103374 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 886/1808] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:47 2022] DELETE /identity/v3/users/7bb90895b6864fe086f56cf6f4ab6172 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:48.108844 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6e4e18-bfa9-4106-8a49-c27aeefe72d5 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:48.109072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6e4e18-bfa9-4106-8a49-c27aeefe72d5 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:48.109361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6e4e18-bfa9-4106-8a49-c27aeefe72d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:48.129886 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-6a6e4e18-bfa9-4106-8a49-c27aeefe72d5 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:48.131270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6a6e4e18-bfa9-4106-8a49-c27aeefe72d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:48.196598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20d3752-24ac-4828-baf5-c97477658545 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:48.196598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20d3752-24ac-4828-baf5-c97477658545 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:48.196598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20d3752-24ac-4828-baf5-c97477658545 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:48.203931 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 923/1809] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:48.215842 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6e4e18-bfa9-4106-8a49-c27aeefe72d5 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:48.217991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6a6e4e18-bfa9-4106-8a49-c27aeefe72d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:48.220827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d20d3752-24ac-4828-baf5-c97477658545 None None] MFA Rules not processed for user `33b136afbf8a4d98a9d62053b68707e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:48.222005 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d20d3752-24ac-4828-baf5-c97477658545 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:48.253315 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:48.283059 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6e4e18-bfa9-4106-8a49-c27aeefe72d5 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:48.284053 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 887/1810] 10.222.0.25 () {58 vars in 967 bytes} [Sun Oct 16 00:58:48 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:48.287434 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a5079e-5a74-4c72-b841-fea50d92d49d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:48.288056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a5079e-5a74-4c72-b841-fea50d92d49d service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:48.288748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a5079e-5a74-4c72-b841-fea50d92d49d service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:48.289127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a5079e-5a74-4c72-b841-fea50d92d49d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:48.297439 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20d3752-24ac-4828-baf5-c97477658545 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:48.298934 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d20d3752-24ac-4828-baf5-c97477658545 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:48.302176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a5079e-5a74-4c72-b841-fea50d92d49d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:48.304327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a5079e-5a74-4c72-b841-fea50d92d49d service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:48.315602 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 924/1811] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:48.388534 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20d3752-24ac-4828-baf5-c97477658545 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:48.395375 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6a857a-3c1f-4fc5-bf4d-bd87bfaa9086 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de64d498da78448394b0d4d1851a2789', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:48.396019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6a857a-3c1f-4fc5-bf4d-bd87bfaa9086 service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:48.396489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6a857a-3c1f-4fc5-bf4d-bd87bfaa9086 service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:48.396753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6a857a-3c1f-4fc5-bf4d-bd87bfaa9086 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:48.406937 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6a857a-3c1f-4fc5-bf4d-bd87bfaa9086 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:48.408719 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6a857a-3c1f-4fc5-bf4d-bd87bfaa9086 service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:48.417700 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 888/1812] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 16 00:58:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 16 00:58:48.513355 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a5079e-5a74-4c72-b841-fea50d92d49d service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:48.514898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-77a5079e-5a74-4c72-b841-fea50d92d49d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:48.532154 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a5079e-5a74-4c72-b841-fea50d92d49d service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:48.542779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f264d13-cc01-4858-a8d7-da6a9329f6d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:48.543510 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f264d13-cc01-4858-a8d7-da6a9329f6d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:48.543798 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f264d13-cc01-4858-a8d7-da6a9329f6d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:48.544076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f264d13-cc01-4858-a8d7-da6a9329f6d3 admin admin] PATH_INFO: `/v3/users/f841074dc67143acafcb151baf963113` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:48.545226 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f264d13-cc01-4858-a8d7-da6a9329f6d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f841074dc67143acafcb151baf963113)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:48.547425 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f264d13-cc01-4858-a8d7-da6a9329f6d3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:48.623279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-6f264d13-cc01-4858-a8d7-da6a9329f6d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f841074dc67143acafcb151baf963113'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:48.623997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-6f264d13-cc01-4858-a8d7-da6a9329f6d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f841074dc67143acafcb151baf963113'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:48.632428 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-6f264d13-cc01-4858-a8d7-da6a9329f6d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f841074dc67143acafcb151baf963113'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:48.655425 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-6f264d13-cc01-4858-a8d7-da6a9329f6d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f841074dc67143acafcb151baf963113'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:48.662688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6a857a-3c1f-4fc5-bf4d-bd87bfaa9086 service swift] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:48.664047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-6f264d13-cc01-4858-a8d7-da6a9329f6d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f841074dc67143acafcb151baf963113'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:48.664287 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7d6a857a-3c1f-4fc5-bf4d-bd87bfaa9086 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:48.670629 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 925/1813] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:48 2022] DELETE /identity/v3/users/f841074dc67143acafcb151baf963113 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:48.675112 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54a8a8e-8cdb-4af5-aba1-2df819989096 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:48.675550 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54a8a8e-8cdb-4af5-aba1-2df819989096 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:48.675835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54a8a8e-8cdb-4af5-aba1-2df819989096 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:48.694493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6a857a-3c1f-4fc5-bf4d-bd87bfaa9086 service swift] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:48.703802 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-f54a8a8e-8cdb-4af5-aba1-2df819989096 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:48.705051 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-f54a8a8e-8cdb-4af5-aba1-2df819989096 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:48.716207 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:48.789722 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:48.790408 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:48.790690 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:48.791059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108 admin admin] PATH_INFO: `/v3/projects/103f30a9d97c4082ba59a10cc3bc218b` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:48.793923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108 admin admin] RBAC: Authorizing `identity:delete_project(project_id=103f30a9d97c4082ba59a10cc3bc218b)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:48.796710 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:48.850908 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 926/1814] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:58:48.865472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54a8a8e-8cdb-4af5-aba1-2df819989096 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:48.868959 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-f54a8a8e-8cdb-4af5-aba1-2df819989096 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:48.892383 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '103f30a9d97c4082ba59a10cc3bc218b', 'request_id': 'req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:48.908196 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '103f30a9d97c4082ba59a10cc3bc218b', 'request_id': 'req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:48.916285 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '103f30a9d97c4082ba59a10cc3bc218b', 'request_id': 'req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:48.950642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108 admin admin] The token cache is being invalidate because project 103f30a9d97c4082ba59a10cc3bc218b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:48.951232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:48.951925 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:48.960701 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 889/1815] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:48 2022] DELETE /identity/v3/projects/103f30a9d97c4082ba59a10cc3bc218b => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:48.963119 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:48.965420 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:48.967695 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:48.985961 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54a8a8e-8cdb-4af5-aba1-2df819989096 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:49.056327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f394d15-8c92-4ef1-8f6f-cdfe2e4642bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:49.057381 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f394d15-8c92-4ef1-8f6f-cdfe2e4642bc service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:49.057803 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f394d15-8c92-4ef1-8f6f-cdfe2e4642bc service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:49.058183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f394d15-8c92-4ef1-8f6f-cdfe2e4642bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:49.060607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6f394d15-8c92-4ef1-8f6f-cdfe2e4642bc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:49.075074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9eb6ccd-e2be-49b5-8b8a-696d3a1a2108 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:49.081069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f394d15-8c92-4ef1-8f6f-cdfe2e4642bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:49.084404 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88aee46-6dd8-4698-830a-318717cfbd04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:49.085044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f394d15-8c92-4ef1-8f6f-cdfe2e4642bc service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:49.085175 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88aee46-6dd8-4698-830a-318717cfbd04 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:49.085369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88aee46-6dd8-4698-830a-318717cfbd04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:49.085647 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88aee46-6dd8-4698-830a-318717cfbd04 admin admin] PATH_INFO: `/v3/users/54f34cb777e648d892a848a44b6df3aa` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:49.091934 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88aee46-6dd8-4698-830a-318717cfbd04 admin admin] RBAC: Authorizing `identity:delete_user(user_id=54f34cb777e648d892a848a44b6df3aa)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:49.096719 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88aee46-6dd8-4698-830a-318717cfbd04 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:49.111478 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 927/1816] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:49.164773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a88aee46-6dd8-4698-830a-318717cfbd04 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '54f34cb777e648d892a848a44b6df3aa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.166114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a88aee46-6dd8-4698-830a-318717cfbd04 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '54f34cb777e648d892a848a44b6df3aa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.172008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:49.178123 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a88aee46-6dd8-4698-830a-318717cfbd04 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '54f34cb777e648d892a848a44b6df3aa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.179565 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a88aee46-6dd8-4698-830a-318717cfbd04 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '54f34cb777e648d892a848a44b6df3aa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.218488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-a88aee46-6dd8-4698-830a-318717cfbd04 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '54f34cb777e648d892a848a44b6df3aa'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.249947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 890/1817] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:48 2022] DELETE /identity/v3/users/54f34cb777e648d892a848a44b6df3aa => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:49.500170 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f394d15-8c92-4ef1-8f6f-cdfe2e4642bc service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:49.501871 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6f394d15-8c92-4ef1-8f6f-cdfe2e4642bc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:49.518092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f394d15-8c92-4ef1-8f6f-cdfe2e4642bc service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:49.609234 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4792a3c6-4b96-4cd5-a68a-33da8bb4c0e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:49.609918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4792a3c6-4b96-4cd5-a68a-33da8bb4c0e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:49.610202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4792a3c6-4b96-4cd5-a68a-33da8bb4c0e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:49.610483 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4792a3c6-4b96-4cd5-a68a-33da8bb4c0e6 admin admin] PATH_INFO: `/v3/projects/d49d0817c6b74a108b6bb1499b6fd482` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:49.611668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4792a3c6-4b96-4cd5-a68a-33da8bb4c0e6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d49d0817c6b74a108b6bb1499b6fd482)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:49.613589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4792a3c6-4b96-4cd5-a68a-33da8bb4c0e6 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:49.617100 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88aee46-6dd8-4698-830a-318717cfbd04 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:49.619830 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a88aee46-6dd8-4698-830a-318717cfbd04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:49.647824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88aee46-6dd8-4698-830a-318717cfbd04 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:49.667576 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:49.671453 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:49.671917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:49.672392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e admin admin] PATH_INFO: `/v3/projects/c4cc37c4e35e4247838082ded1a71a2c` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:49.673718 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e admin admin] RBAC: Authorizing `identity:delete_project(project_id=c4cc37c4e35e4247838082ded1a71a2c)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:49.676008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:49.716774 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4792a3c6-4b96-4cd5-a68a-33da8bb4c0e6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd49d0817c6b74a108b6bb1499b6fd482', 'request_id': 'req-4792a3c6-4b96-4cd5-a68a-33da8bb4c0e6'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.727592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4792a3c6-4b96-4cd5-a68a-33da8bb4c0e6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd49d0817c6b74a108b6bb1499b6fd482', 'request_id': 'req-4792a3c6-4b96-4cd5-a68a-33da8bb4c0e6'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.742612 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4792a3c6-4b96-4cd5-a68a-33da8bb4c0e6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd49d0817c6b74a108b6bb1499b6fd482', 'request_id': 'req-4792a3c6-4b96-4cd5-a68a-33da8bb4c0e6'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.747991 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c4cc37c4e35e4247838082ded1a71a2c', 'request_id': 'req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.756159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4792a3c6-4b96-4cd5-a68a-33da8bb4c0e6 admin admin] The token cache is being invalidate because project d49d0817c6b74a108b6bb1499b6fd482 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:49.756567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4792a3c6-4b96-4cd5-a68a-33da8bb4c0e6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.757477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4792a3c6-4b96-4cd5-a68a-33da8bb4c0e6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.758239 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c4cc37c4e35e4247838082ded1a71a2c', 'request_id': 'req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.759406 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 928/1818] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:49 2022] DELETE /identity/v3/projects/d49d0817c6b74a108b6bb1499b6fd482 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:49.765681 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa036177-205d-442e-a3a3-8379505dd4e6 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:49.767392 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa036177-205d-442e-a3a3-8379505dd4e6 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:49.767668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa036177-205d-442e-a3a3-8379505dd4e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:49.774626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c4cc37c4e35e4247838082ded1a71a2c', 'request_id': 'req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.787388 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e admin admin] The token cache is being invalidate because project c4cc37c4e35e4247838082ded1a71a2c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:49.787683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.790229 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.797755 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-aa036177-205d-442e-a3a3-8379505dd4e6 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:49.797755 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-aa036177-205d-442e-a3a3-8379505dd4e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:49.800243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 891/1819] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:49 2022] DELETE /identity/v3/projects/c4cc37c4e35e4247838082ded1a71a2c => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:58:49.807430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:49.815457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:49.844697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:49.894070 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 929/1820] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:49.902547 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9bfa9a-fd02-4430-8f00-289ce3b19d80 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:49.902799 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9bfa9a-fd02-4430-8f00-289ce3b19d80 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:49.903106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9bfa9a-fd02-4430-8f00-289ce3b19d80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:49.917980 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de94cc87-1f8e-4bea-a6b5-b8c097c02c6e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:49.922628 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de222339-6769-460e-b4b4-88135124c35b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:49.923330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de222339-6769-460e-b4b4-88135124c35b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:49.923718 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de222339-6769-460e-b4b4-88135124c35b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:49.924031 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de222339-6769-460e-b4b4-88135124c35b admin admin] PATH_INFO: `/v3/users/18ad8f619c6a4955870413995be857ba` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:49.924196 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-2e9bfa9a-fd02-4430-8f00-289ce3b19d80 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:49.925214 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de222339-6769-460e-b4b4-88135124c35b admin admin] RBAC: Authorizing `identity:delete_user(user_id=18ad8f619c6a4955870413995be857ba)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:49.925503 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-2e9bfa9a-fd02-4430-8f00-289ce3b19d80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:49.927532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de222339-6769-460e-b4b4-88135124c35b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:49.938413 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 930/1821] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:58:49.942660 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9bfa9a-fd02-4430-8f00-289ce3b19d80 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:49.953811 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9bfa9a-fd02-4430-8f00-289ce3b19d80 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:49.960778 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78467b03-3462-4ec7-ac7c-8e531423d4b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:49.961457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78467b03-3462-4ec7-ac7c-8e531423d4b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:49.961718 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78467b03-3462-4ec7-ac7c-8e531423d4b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:49.961972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78467b03-3462-4ec7-ac7c-8e531423d4b3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:49.962731 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78467b03-3462-4ec7-ac7c-8e531423d4b3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:58:49.963095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78467b03-3462-4ec7-ac7c-8e531423d4b3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:49.965597 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78467b03-3462-4ec7-ac7c-8e531423d4b3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:49.978767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-de222339-6769-460e-b4b4-88135124c35b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '18ad8f619c6a4955870413995be857ba'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.979004 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 931/1822] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:58:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:58:49.979738 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-de222339-6769-460e-b4b4-88135124c35b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '18ad8f619c6a4955870413995be857ba'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.983704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78467b03-3462-4ec7-ac7c-8e531423d4b3 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:49.984295 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:DELETE=18,keystone:SELECT=80,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:49.985596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-78467b03-3462-4ec7-ac7c-8e531423d4b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:49.991616 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-de222339-6769-460e-b4b4-88135124c35b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '18ad8f619c6a4955870413995be857ba'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:49.992632 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-de222339-6769-460e-b4b4-88135124c35b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '18ad8f619c6a4955870413995be857ba'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:50.000391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78467b03-3462-4ec7-ac7c-8e531423d4b3 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:50.007338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d290ee9-9720-458a-940c-b302fd5b8e8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:50.008019 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d290ee9-9720-458a-940c-b302fd5b8e8f admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:50.008312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d290ee9-9720-458a-940c-b302fd5b8e8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:50.008623 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d290ee9-9720-458a-940c-b302fd5b8e8f admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:50.008766 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-de222339-6769-460e-b4b4-88135124c35b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '18ad8f619c6a4955870413995be857ba'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:50.009765 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d290ee9-9720-458a-940c-b302fd5b8e8f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:50.013510 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d290ee9-9720-458a-940c-b302fd5b8e8f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:50.026992 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 892/1823] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:49 2022] DELETE /identity/v3/users/18ad8f619c6a4955870413995be857ba => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:50.046415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 932/1824] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:49 2022] POST /identity/v3/projects => generated 385 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:58:50.050534 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de222339-6769-460e-b4b4-88135124c35b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:50.167676 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de222339-6769-460e-b4b4-88135124c35b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:50.242596 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fcb485-5a05-4366-9679-74088c16d165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:50.243359 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fcb485-5a05-4366-9679-74088c16d165 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:50.243629 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fcb485-5a05-4366-9679-74088c16d165 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:50.243986 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fcb485-5a05-4366-9679-74088c16d165 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:50.247919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fcb485-5a05-4366-9679-74088c16d165 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:50.250316 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fcb485-5a05-4366-9679-74088c16d165 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:50.300104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 893/1825] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:50 2022] POST /identity/v3/users => generated 293 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:50.304722 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d290ee9-9720-458a-940c-b302fd5b8e8f admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:50.317836 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d290ee9-9720-458a-940c-b302fd5b8e8f admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:50.323097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686dc821-4ce0-4e4a-b43a-7de4a4ae0832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:50.324466 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686dc821-4ce0-4e4a-b43a-7de4a4ae0832 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:50.324466 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686dc821-4ce0-4e4a-b43a-7de4a4ae0832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:50.324671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686dc821-4ce0-4e4a-b43a-7de4a4ae0832 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:50.325256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686dc821-4ce0-4e4a-b43a-7de4a4ae0832 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:50.327180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686dc821-4ce0-4e4a-b43a-7de4a4ae0832 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:50.342451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 933/1826] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:50 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:50.346282 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fcb485-5a05-4366-9679-74088c16d165 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:50.359995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fcb485-5a05-4366-9679-74088c16d165 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:50.367269 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f4ba0f-b317-459c-8cd4-3a76fea196dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:50.368002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f4ba0f-b317-459c-8cd4-3a76fea196dd admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:50.368267 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f4ba0f-b317-459c-8cd4-3a76fea196dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:50.368565 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f4ba0f-b317-459c-8cd4-3a76fea196dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:50.369253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f4ba0f-b317-459c-8cd4-3a76fea196dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:50.371066 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f4ba0f-b317-459c-8cd4-3a76fea196dd admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:50.383994 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 894/1827] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:50 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:50.388965 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686dc821-4ce0-4e4a-b43a-7de4a4ae0832 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:50.403561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686dc821-4ce0-4e4a-b43a-7de4a4ae0832 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:50.412692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912c02bd-6414-4fb7-9e95-4798a4ac54aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:50.413979 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912c02bd-6414-4fb7-9e95-4798a4ac54aa admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:50.414267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912c02bd-6414-4fb7-9e95-4798a4ac54aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:50.414540 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912c02bd-6414-4fb7-9e95-4798a4ac54aa admin admin] PATH_INFO: `/v3/projects/5f810e546f6641cabdf5eb3ba3469c6b/users/445cb44c97724c618ae9c636c074f0ee/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:50.422852 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f4ba0f-b317-459c-8cd4-3a76fea196dd admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:50.425899 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-10f4ba0f-b317-459c-8cd4-3a76fea196dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:50.442640 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912c02bd-6414-4fb7-9e95-4798a4ac54aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f810e546f6641cabdf5eb3ba3469c6b, user_id=445cb44c97724c618ae9c636c074f0ee, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:50.445675 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912c02bd-6414-4fb7-9e95-4798a4ac54aa admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:50.451918 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f4ba0f-b317-459c-8cd4-3a76fea196dd admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:50.469416 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83591def-e89d-4b30-9eb3-161eae088e0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:50.470275 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 934/1828] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:50 2022] PUT /identity/v3/projects/5f810e546f6641cabdf5eb3ba3469c6b/users/445cb44c97724c618ae9c636c074f0ee/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:50.475500 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83591def-e89d-4b30-9eb3-161eae088e0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:50.475802 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83591def-e89d-4b30-9eb3-161eae088e0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:50.476114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83591def-e89d-4b30-9eb3-161eae088e0c admin admin] PATH_INFO: `/v3/projects/0256e137c4ef4828823b77ceb13e5504` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:50.477482 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83591def-e89d-4b30-9eb3-161eae088e0c admin admin] RBAC: Authorizing `identity:delete_project(project_id=0256e137c4ef4828823b77ceb13e5504)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:50.479589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83591def-e89d-4b30-9eb3-161eae088e0c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:50.481468 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd7a6e7-67a8-4b6c-ba37-c43d752f106c None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:50.481764 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd7a6e7-67a8-4b6c-ba37-c43d752f106c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:50.482047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd7a6e7-67a8-4b6c-ba37-c43d752f106c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:50.519666 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-2cd7a6e7-67a8-4b6c-ba37-c43d752f106c None None] MFA Rules not processed for user `445cb44c97724c618ae9c636c074f0ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:50.520899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-2cd7a6e7-67a8-4b6c-ba37-c43d752f106c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:50.537991 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-83591def-e89d-4b30-9eb3-161eae088e0c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0256e137c4ef4828823b77ceb13e5504', 'request_id': 'req-83591def-e89d-4b30-9eb3-161eae088e0c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:50.540290 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:DELETE=6,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:50.547820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-83591def-e89d-4b30-9eb3-161eae088e0c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0256e137c4ef4828823b77ceb13e5504', 'request_id': 'req-83591def-e89d-4b30-9eb3-161eae088e0c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:50.556470 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-83591def-e89d-4b30-9eb3-161eae088e0c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0256e137c4ef4828823b77ceb13e5504', 'request_id': 'req-83591def-e89d-4b30-9eb3-161eae088e0c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:50.575367 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-83591def-e89d-4b30-9eb3-161eae088e0c admin admin] The token cache is being invalidate because project 0256e137c4ef4828823b77ceb13e5504 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:50.575661 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-83591def-e89d-4b30-9eb3-161eae088e0c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:50.576603 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-83591def-e89d-4b30-9eb3-161eae088e0c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:50.578752 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 895/1829] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:50 2022] DELETE /identity/v3/projects/0256e137c4ef4828823b77ceb13e5504 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:50.584862 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f19e98-f547-429a-8ca0-6ef597216184 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:50.584862 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f19e98-f547-429a-8ca0-6ef597216184 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:50.585100 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f19e98-f547-429a-8ca0-6ef597216184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:50.628543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-44f19e98-f547-429a-8ca0-6ef597216184 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:50.630530 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-44f19e98-f547-429a-8ca0-6ef597216184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:50.645168 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 935/1830] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:50 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:58:50.649401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd7a6e7-67a8-4b6c-ba37-c43d752f106c None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:50.650936 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-2cd7a6e7-67a8-4b6c-ba37-c43d752f106c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:50.716028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:50.716749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 896/1831] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:50.725064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7977d7e-7013-41b7-92c2-879d033c89aa None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:50.725328 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7977d7e-7013-41b7-92c2-879d033c89aa None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:50.725603 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7977d7e-7013-41b7-92c2-879d033c89aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:50.736724 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd7a6e7-67a8-4b6c-ba37-c43d752f106c None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:50.741901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7358ebf-7388-4151-91fb-667d16d7a999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:50.742526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7358ebf-7388-4151-91fb-667d16d7a999 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:50.742807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7358ebf-7388-4151-91fb-667d16d7a999 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:50.743113 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7358ebf-7388-4151-91fb-667d16d7a999 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:50.743879 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7358ebf-7388-4151-91fb-667d16d7a999 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:50.746455 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7358ebf-7388-4151-91fb-667d16d7a999 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:50.746609 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f7977d7e-7013-41b7-92c2-879d033c89aa None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:50.747867 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f7977d7e-7013-41b7-92c2-879d033c89aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:50.756094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 897/1832] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:58:50.760298 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7977d7e-7013-41b7-92c2-879d033c89aa None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:50.770982 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7977d7e-7013-41b7-92c2-879d033c89aa None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:50.778600 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 936/1833] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:50 2022] POST /identity/v3/projects => generated 385 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:50.779382 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14182f38-1f2d-4e9d-9de7-60329501bb22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:50.780002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14182f38-1f2d-4e9d-9de7-60329501bb22 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:50.780282 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14182f38-1f2d-4e9d-9de7-60329501bb22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:50.780556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14182f38-1f2d-4e9d-9de7-60329501bb22 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:50.781208 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14182f38-1f2d-4e9d-9de7-60329501bb22 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:58:50.781514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14182f38-1f2d-4e9d-9de7-60329501bb22 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:50.783158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14182f38-1f2d-4e9d-9de7-60329501bb22 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:50.783857 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7358ebf-7388-4151-91fb-667d16d7a999 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:50.799401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 898/1834] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:58:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 00:58:50.800483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14182f38-1f2d-4e9d-9de7-60329501bb22 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:50.810998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14182f38-1f2d-4e9d-9de7-60329501bb22 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:50.851774 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7358ebf-7388-4151-91fb-667d16d7a999 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:50.858063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6e1250-5815-42b0-b6fe-563fcd212ddd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:50.858751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6e1250-5815-42b0-b6fe-563fcd212ddd admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:50.859057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6e1250-5815-42b0-b6fe-563fcd212ddd admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:50.859304 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6e1250-5815-42b0-b6fe-563fcd212ddd admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:50.859957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6e1250-5815-42b0-b6fe-563fcd212ddd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:50.861934 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6e1250-5815-42b0-b6fe-563fcd212ddd admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:50.884062 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b0abdf-7494-413c-8791-e0572ade49fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:50.884701 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b0abdf-7494-413c-8791-e0572ade49fc admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:50.884968 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b0abdf-7494-413c-8791-e0572ade49fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:50.885256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b0abdf-7494-413c-8791-e0572ade49fc admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:50.887711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b0abdf-7494-413c-8791-e0572ade49fc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:50.888347 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b0abdf-7494-413c-8791-e0572ade49fc admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:50.909430 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 937/1835] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:50 2022] POST /identity/v3/users => generated 293 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:58:50.915452 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6e1250-5815-42b0-b6fe-563fcd212ddd admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:50.929711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 899/1836] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:50 2022] POST /identity/v3/projects => generated 381 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:50.930667 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b0abdf-7494-413c-8791-e0572ade49fc admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:50.932608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-11b0abdf-7494-413c-8791-e0572ade49fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:50.941051 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6e1250-5815-42b0-b6fe-563fcd212ddd admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:50.988245 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:INSERT=2,keystone:DELETE=3,keystone:SELECT=95 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:51.007308 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196db500-1fd0-4f55-8cb5-7e447a552f45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:51.008073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196db500-1fd0-4f55-8cb5-7e447a552f45 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.008458 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196db500-1fd0-4f55-8cb5-7e447a552f45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.008802 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196db500-1fd0-4f55-8cb5-7e447a552f45 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.009522 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196db500-1fd0-4f55-8cb5-7e447a552f45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:51.010286 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b0abdf-7494-413c-8791-e0572ade49fc admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:51.011367 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196db500-1fd0-4f55-8cb5-7e447a552f45 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:51.015937 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08db2bca-2054-4fea-9c81-1be3aae4d05d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:51.016741 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08db2bca-2054-4fea-9c81-1be3aae4d05d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.017027 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08db2bca-2054-4fea-9c81-1be3aae4d05d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.017303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08db2bca-2054-4fea-9c81-1be3aae4d05d admin admin] PATH_INFO: `/v3/users/86fccabd87a84389b219b00cdcc857e2` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.018678 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08db2bca-2054-4fea-9c81-1be3aae4d05d admin admin] RBAC: Authorizing `identity:delete_user(user_id=86fccabd87a84389b219b00cdcc857e2)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:51.021065 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08db2bca-2054-4fea-9c81-1be3aae4d05d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:51.022513 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 938/1837] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:50 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:51.023546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196db500-1fd0-4f55-8cb5-7e447a552f45 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:51.034216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196db500-1fd0-4f55-8cb5-7e447a552f45 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:51.039032 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690bd246-41b0-4cb2-a217-6cabfa0ad42b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:51.039685 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690bd246-41b0-4cb2-a217-6cabfa0ad42b admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.039927 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690bd246-41b0-4cb2-a217-6cabfa0ad42b admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.040194 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690bd246-41b0-4cb2-a217-6cabfa0ad42b admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.040826 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690bd246-41b0-4cb2-a217-6cabfa0ad42b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:51.043518 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690bd246-41b0-4cb2-a217-6cabfa0ad42b admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:51.069714 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 939/1838] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:51 2022] POST /identity/v3/users => generated 291 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:51.070557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690bd246-41b0-4cb2-a217-6cabfa0ad42b admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:51.078605 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-08db2bca-2054-4fea-9c81-1be3aae4d05d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '86fccabd87a84389b219b00cdcc857e2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:51.079431 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-08db2bca-2054-4fea-9c81-1be3aae4d05d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '86fccabd87a84389b219b00cdcc857e2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:51.080598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690bd246-41b0-4cb2-a217-6cabfa0ad42b admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:51.086109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722b6085-8e15-48b4-b6c7-7cb9f265d9dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:51.086706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722b6085-8e15-48b4-b6c7-7cb9f265d9dd admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.087202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722b6085-8e15-48b4-b6c7-7cb9f265d9dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.087478 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-08db2bca-2054-4fea-9c81-1be3aae4d05d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '86fccabd87a84389b219b00cdcc857e2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:51.087777 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722b6085-8e15-48b4-b6c7-7cb9f265d9dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.088249 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722b6085-8e15-48b4-b6c7-7cb9f265d9dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:51.088397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-08db2bca-2054-4fea-9c81-1be3aae4d05d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '86fccabd87a84389b219b00cdcc857e2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:51.090475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722b6085-8e15-48b4-b6c7-7cb9f265d9dd admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:51.103325 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 940/1839] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:51 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:51.105415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-08db2bca-2054-4fea-9c81-1be3aae4d05d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '86fccabd87a84389b219b00cdcc857e2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:51.109120 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722b6085-8e15-48b4-b6c7-7cb9f265d9dd admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:51.112469 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-722b6085-8e15-48b4-b6c7-7cb9f265d9dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:51.129722 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 900/1840] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:50 2022] DELETE /identity/v3/users/86fccabd87a84389b219b00cdcc857e2 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:58:51.129722 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08db2bca-2054-4fea-9c81-1be3aae4d05d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:51.129722 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-08db2bca-2054-4fea-9c81-1be3aae4d05d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:51.131189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722b6085-8e15-48b4-b6c7-7cb9f265d9dd admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:51.162175 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:51.199790 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7f4463-309c-4a92-a03c-1900b2e4ac18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:51.200418 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7f4463-309c-4a92-a03c-1900b2e4ac18 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.200967 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7f4463-309c-4a92-a03c-1900b2e4ac18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.201094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08db2bca-2054-4fea-9c81-1be3aae4d05d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:51.201237 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7f4463-309c-4a92-a03c-1900b2e4ac18 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.201883 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7f4463-309c-4a92-a03c-1900b2e4ac18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:51.203529 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7f4463-309c-4a92-a03c-1900b2e4ac18 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:51.205940 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79507bb-020f-4db5-90d0-3434f391820b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:51.206792 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79507bb-020f-4db5-90d0-3434f391820b admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.207266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79507bb-020f-4db5-90d0-3434f391820b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.207572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79507bb-020f-4db5-90d0-3434f391820b admin admin] PATH_INFO: `/v3/projects/6d3280811ab24bc884551d5446b5787b/users/ee8c3d18b31646548bc4c82422bd6f4c/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.215859 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 941/1841] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:51 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:51.217792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbe4568-14d3-4d9c-983f-bcf844cdcdbf None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.217792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbe4568-14d3-4d9c-983f-bcf844cdcdbf None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.217792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbe4568-14d3-4d9c-983f-bcf844cdcdbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.225110 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79507bb-020f-4db5-90d0-3434f391820b admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d3280811ab24bc884551d5446b5787b, user_id=ee8c3d18b31646548bc4c82422bd6f4c, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:51.227908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79507bb-020f-4db5-90d0-3434f391820b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:51.238621 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-3fbe4568-14d3-4d9c-983f-bcf844cdcdbf None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:51.239798 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-3fbe4568-14d3-4d9c-983f-bcf844cdcdbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:51.241294 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 901/1842] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:51 2022] PUT /identity/v3/projects/6d3280811ab24bc884551d5446b5787b/users/ee8c3d18b31646548bc4c82422bd6f4c/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:51.242107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79507bb-020f-4db5-90d0-3434f391820b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:51.294622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:51.312657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79507bb-020f-4db5-90d0-3434f391820b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:51.316678 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 942/1843] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:51.318632 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26312c94-29d4-4f5c-ae48-2c6e35b2e130 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.319023 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26312c94-29d4-4f5c-ae48-2c6e35b2e130 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.319394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26312c94-29d4-4f5c-ae48-2c6e35b2e130 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.326850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a46769-cb80-4d59-9d20-da9f1631d3e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:51.327729 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a46769-cb80-4d59-9d20-da9f1631d3e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.328188 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a46769-cb80-4d59-9d20-da9f1631d3e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.328635 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a46769-cb80-4d59-9d20-da9f1631d3e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.329299 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a46769-cb80-4d59-9d20-da9f1631d3e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:51.330998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a46769-cb80-4d59-9d20-da9f1631d3e5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:51.343529 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 902/1844] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:51 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:51.343743 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a46769-cb80-4d59-9d20-da9f1631d3e5 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:51.352146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-35a46769-cb80-4d59-9d20-da9f1631d3e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:51.352973 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-26312c94-29d4-4f5c-ae48-2c6e35b2e130 None None] MFA Rules not processed for user `ee8c3d18b31646548bc4c82422bd6f4c`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:51.354353 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-26312c94-29d4-4f5c-ae48-2c6e35b2e130 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:51.429516 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a46769-cb80-4d59-9d20-da9f1631d3e5 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:51.435657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f76d7d-51d6-44d8-a1c1-4d0208f42acd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:51.436401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f76d7d-51d6-44d8-a1c1-4d0208f42acd service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.436684 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f76d7d-51d6-44d8-a1c1-4d0208f42acd service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.436967 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f76d7d-51d6-44d8-a1c1-4d0208f42acd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.451316 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f76d7d-51d6-44d8-a1c1-4d0208f42acd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:51.452794 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f76d7d-51d6-44d8-a1c1-4d0208f42acd service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:51.458432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 943/1845] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:51 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:51.463037 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26312c94-29d4-4f5c-ae48-2c6e35b2e130 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:51.463350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 903/1846] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:51.464882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f76d7d-51d6-44d8-a1c1-4d0208f42acd service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:51.475661 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f76d7d-51d6-44d8-a1c1-4d0208f42acd service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:51.479065 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26312c94-29d4-4f5c-ae48-2c6e35b2e130 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:51.480757 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0817fba5-d6bf-419d-be46-69ffd39deba9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:51.481386 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0817fba5-d6bf-419d-be46-69ffd39deba9 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.481834 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0817fba5-d6bf-419d-be46-69ffd39deba9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.482113 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0817fba5-d6bf-419d-be46-69ffd39deba9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.482756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0817fba5-d6bf-419d-be46-69ffd39deba9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:51.484727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0817fba5-d6bf-419d-be46-69ffd39deba9 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:51.488754 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e78c4a1-f556-4c91-b431-07182e3abf6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:51.489667 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e78c4a1-f556-4c91-b431-07182e3abf6e admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.490184 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e78c4a1-f556-4c91-b431-07182e3abf6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.490801 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e78c4a1-f556-4c91-b431-07182e3abf6e admin admin] PATH_INFO: `/v3/projects/5073a7a54615493b8587ad0062b8b7a6/users/b42f4237a09245d3a3df94ff8ae02526/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.508208 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e78c4a1-f556-4c91-b431-07182e3abf6e admin admin] RBAC: Authorizing `identity:create_grant(project_id=5073a7a54615493b8587ad0062b8b7a6, user_id=b42f4237a09245d3a3df94ff8ae02526, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:51.515623 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e78c4a1-f556-4c91-b431-07182e3abf6e admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:51.521546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 904/1847] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:51 2022] POST /identity/v3/projects => generated 383 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:51.525839 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0817fba5-d6bf-419d-be46-69ffd39deba9 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:51.534807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 944/1848] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:51 2022] PUT /identity/v3/projects/5073a7a54615493b8587ad0062b8b7a6/users/b42f4237a09245d3a3df94ff8ae02526/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:51.547445 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1961e249-74e9-47f4-b3e4-315de3cc4e14 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.547730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1961e249-74e9-47f4-b3e4-315de3cc4e14 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.548044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1961e249-74e9-47f4-b3e4-315de3cc4e14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.579871 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-1961e249-74e9-47f4-b3e4-315de3cc4e14 None None] MFA Rules not processed for user `b42f4237a09245d3a3df94ff8ae02526`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:51.581280 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-1961e249-74e9-47f4-b3e4-315de3cc4e14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:51.603780 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:51.616237 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0817fba5-d6bf-419d-be46-69ffd39deba9 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:51.683325 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 945/1849] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:51 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:51.683325 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1961e249-74e9-47f4-b3e4-315de3cc4e14 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:51.715352 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88accce-1110-4622-8986-50e8d61d011c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:51.716054 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88accce-1110-4622-8986-50e8d61d011c admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.716240 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88accce-1110-4622-8986-50e8d61d011c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.716510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88accce-1110-4622-8986-50e8d61d011c admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.717143 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88accce-1110-4622-8986-50e8d61d011c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:51.719095 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88accce-1110-4622-8986-50e8d61d011c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:51.739316 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:51.747397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 905/1850] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:51 2022] POST /identity/v3/users => generated 298 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:51.749793 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88accce-1110-4622-8986-50e8d61d011c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:51.753786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d88accce-1110-4622-8986-50e8d61d011c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:51.779762 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88accce-1110-4622-8986-50e8d61d011c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:51.787619 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73406047-c587-46d3-84b2-f31482f7d7c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:51.788510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73406047-c587-46d3-84b2-f31482f7d7c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.788790 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73406047-c587-46d3-84b2-f31482f7d7c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.789061 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73406047-c587-46d3-84b2-f31482f7d7c5 admin admin] PATH_INFO: `/v3/projects/39d1c20705604f8794fb441271e5afd3` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.790182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73406047-c587-46d3-84b2-f31482f7d7c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=39d1c20705604f8794fb441271e5afd3)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:51.792422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73406047-c587-46d3-84b2-f31482f7d7c5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:51.798622 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1961e249-74e9-47f4-b3e4-315de3cc4e14 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:51.804115 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6575bee-8901-4a00-87a5-fe777038c79f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:51.804735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6575bee-8901-4a00-87a5-fe777038c79f admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.805039 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6575bee-8901-4a00-87a5-fe777038c79f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.805315 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6575bee-8901-4a00-87a5-fe777038c79f admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.805953 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6575bee-8901-4a00-87a5-fe777038c79f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:51.808986 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6575bee-8901-4a00-87a5-fe777038c79f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:51.845392 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 946/1851] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:51 2022] POST /identity/v3/projects => generated 381 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:51.845392 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6575bee-8901-4a00-87a5-fe777038c79f admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:51.854222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-73406047-c587-46d3-84b2-f31482f7d7c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '39d1c20705604f8794fb441271e5afd3', 'request_id': 'req-73406047-c587-46d3-84b2-f31482f7d7c5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:51.862545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-73406047-c587-46d3-84b2-f31482f7d7c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '39d1c20705604f8794fb441271e5afd3', 'request_id': 'req-73406047-c587-46d3-84b2-f31482f7d7c5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:51.874317 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-73406047-c587-46d3-84b2-f31482f7d7c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '39d1c20705604f8794fb441271e5afd3', 'request_id': 'req-73406047-c587-46d3-84b2-f31482f7d7c5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:51.891355 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-73406047-c587-46d3-84b2-f31482f7d7c5 admin admin] The token cache is being invalidate because project 39d1c20705604f8794fb441271e5afd3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:51.891653 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-73406047-c587-46d3-84b2-f31482f7d7c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:51.892651 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-73406047-c587-46d3-84b2-f31482f7d7c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:51.894344 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 906/1852] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:51 2022] DELETE /identity/v3/projects/39d1c20705604f8794fb441271e5afd3 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:51.895417 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73406047-c587-46d3-84b2-f31482f7d7c5 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:51.896857 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-73406047-c587-46d3-84b2-f31482f7d7c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:51.942565 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6575bee-8901-4a00-87a5-fe777038c79f admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:51.948776 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffd5d42-fe05-4fd7-bda0-1716eb080e0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:51.949487 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffd5d42-fe05-4fd7-bda0-1716eb080e0f admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.949860 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffd5d42-fe05-4fd7-bda0-1716eb080e0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.950208 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffd5d42-fe05-4fd7-bda0-1716eb080e0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.951224 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffd5d42-fe05-4fd7-bda0-1716eb080e0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:51.953216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffd5d42-fe05-4fd7-bda0-1716eb080e0f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:51.971891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 947/1853] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:51 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:51.973165 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4854ea65-eb8c-4896-8573-fcf2fe86ac8b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:51.973582 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4854ea65-eb8c-4896-8573-fcf2fe86ac8b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:51.973966 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4854ea65-eb8c-4896-8573-fcf2fe86ac8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:51.996582 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73406047-c587-46d3-84b2-f31482f7d7c5 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:51.998496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-4854ea65-eb8c-4896-8573-fcf2fe86ac8b None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:51.999780 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4854ea65-eb8c-4896-8573-fcf2fe86ac8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:52.001829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6850f0-a74b-414a-bd85-7a1219625ef9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:52.002455 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6850f0-a74b-414a-bd85-7a1219625ef9 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.003027 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6850f0-a74b-414a-bd85-7a1219625ef9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.003393 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6850f0-a74b-414a-bd85-7a1219625ef9 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.004292 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6850f0-a74b-414a-bd85-7a1219625ef9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:52.006786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6850f0-a74b-414a-bd85-7a1219625ef9 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:52.008762 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 948/1854] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:52.009669 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4854ea65-eb8c-4896-8573-fcf2fe86ac8b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:52.011345 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4854ea65-eb8c-4896-8573-fcf2fe86ac8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:52.025473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4854ea65-eb8c-4896-8573-fcf2fe86ac8b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:52.031144 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed6c19f-bce0-46ef-b1ed-a7b4ccd90681 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:52.035034 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 907/1855] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:51 2022] POST /identity/v3/users => generated 297 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:52.035221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed6c19f-bce0-46ef-b1ed-a7b4ccd90681 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.035221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed6c19f-bce0-46ef-b1ed-a7b4ccd90681 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.035221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed6c19f-bce0-46ef-b1ed-a7b4ccd90681 admin admin] PATH_INFO: `/v3/projects/dec347208ed246f9a7c569f90c46f9ac/users/88c250889ad947658bb7bdd72316a75a/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.035493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f19757-024a-467e-ba36-48f940933743 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.035493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f19757-024a-467e-ba36-48f940933743 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.035493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f19757-024a-467e-ba36-48f940933743 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.052590 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed6c19f-bce0-46ef-b1ed-a7b4ccd90681 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dec347208ed246f9a7c569f90c46f9ac, user_id=88c250889ad947658bb7bdd72316a75a, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:52.055717 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed6c19f-bce0-46ef-b1ed-a7b4ccd90681 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:52.063654 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-95f19757-024a-467e-ba36-48f940933743 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:52.065071 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-95f19757-024a-467e-ba36-48f940933743 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:52.076821 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 949/1856] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:52 2022] PUT /identity/v3/projects/dec347208ed246f9a7c569f90c46f9ac/users/88c250889ad947658bb7bdd72316a75a/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:52.079999 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 908/1857] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:52.079999 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f19757-024a-467e-ba36-48f940933743 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:52.081331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed6c19f-bce0-46ef-b1ed-a7b4ccd90681 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:52.090296 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f19757-024a-467e-ba36-48f940933743 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:52.097974 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:52.187509 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed6c19f-bce0-46ef-b1ed-a7b4ccd90681 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:52.197799 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2befd962-01fc-403e-92a6-506c7ce5db55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:52.198669 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2befd962-01fc-403e-92a6-506c7ce5db55 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.199007 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2befd962-01fc-403e-92a6-506c7ce5db55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.199270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2befd962-01fc-403e-92a6-506c7ce5db55 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.200063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2befd962-01fc-403e-92a6-506c7ce5db55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:52.202014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2befd962-01fc-403e-92a6-506c7ce5db55 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:52.205341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44a2088-8646-49a5-975b-be78a114a38b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:52.205971 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44a2088-8646-49a5-975b-be78a114a38b admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.206251 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44a2088-8646-49a5-975b-be78a114a38b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.206520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44a2088-8646-49a5-975b-be78a114a38b admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.207255 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44a2088-8646-49a5-975b-be78a114a38b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:52.209531 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44a2088-8646-49a5-975b-be78a114a38b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:52.213467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 950/1858] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:52 2022] GET /identity/v3/roles => generated 1420 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:52.215725 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2befd962-01fc-403e-92a6-506c7ce5db55 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:52.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 909/1859] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:52 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:52.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44a2088-8646-49a5-975b-be78a114a38b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:52.225996 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2befd962-01fc-403e-92a6-506c7ce5db55 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:52.227593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:52.233225 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787fc16a-0da7-4bce-94b9-e9b0de42efe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:52.233856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787fc16a-0da7-4bce-94b9-e9b0de42efe5 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.234196 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787fc16a-0da7-4bce-94b9-e9b0de42efe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.234477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787fc16a-0da7-4bce-94b9-e9b0de42efe5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.235171 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787fc16a-0da7-4bce-94b9-e9b0de42efe5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:58:52.235461 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44a2088-8646-49a5-975b-be78a114a38b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:52.235565 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787fc16a-0da7-4bce-94b9-e9b0de42efe5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:52.237173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787fc16a-0da7-4bce-94b9-e9b0de42efe5 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:52.243514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced01731-1752-4e40-b54b-7e3cf80550a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:52.244401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced01731-1752-4e40-b54b-7e3cf80550a2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.244830 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced01731-1752-4e40-b54b-7e3cf80550a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.247056 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced01731-1752-4e40-b54b-7e3cf80550a2 admin admin] PATH_INFO: `/v3/projects/dec347208ed246f9a7c569f90c46f9ac/users/88c250889ad947658bb7bdd72316a75a/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.249800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced01731-1752-4e40-b54b-7e3cf80550a2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dec347208ed246f9a7c569f90c46f9ac, user_id=88c250889ad947658bb7bdd72316a75a, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:52.255649 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 951/1860] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:58:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:58:52.255791 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced01731-1752-4e40-b54b-7e3cf80550a2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:52.255883 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787fc16a-0da7-4bce-94b9-e9b0de42efe5 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:52.264296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787fc16a-0da7-4bce-94b9-e9b0de42efe5 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:52.265601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 910/1861] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:52 2022] PUT /identity/v3/projects/dec347208ed246f9a7c569f90c46f9ac/users/88c250889ad947658bb7bdd72316a75a/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:52.266424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced01731-1752-4e40-b54b-7e3cf80550a2 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:52.333384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca837595-5507-4f9b-a31d-c7e38b27b19a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:52.334084 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca837595-5507-4f9b-a31d-c7e38b27b19a admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.334388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca837595-5507-4f9b-a31d-c7e38b27b19a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.334662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca837595-5507-4f9b-a31d-c7e38b27b19a admin admin] PATH_INFO: `/v3/projects/3199ddb07797466cb5d88adf25a33d2c/users/41ecac1fa93740c6bccc370105eb26b8/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.336969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced01731-1752-4e40-b54b-7e3cf80550a2 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:52.342720 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1d0936-947a-41f7-87ea-0403bbf13e9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:52.343470 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1d0936-947a-41f7-87ea-0403bbf13e9b admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.343751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1d0936-947a-41f7-87ea-0403bbf13e9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.344033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1d0936-947a-41f7-87ea-0403bbf13e9b admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.344679 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1d0936-947a-41f7-87ea-0403bbf13e9b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:52.346630 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1d0936-947a-41f7-87ea-0403bbf13e9b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:52.350172 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca837595-5507-4f9b-a31d-c7e38b27b19a admin admin] RBAC: Authorizing `identity:create_grant(project_id=3199ddb07797466cb5d88adf25a33d2c, user_id=41ecac1fa93740c6bccc370105eb26b8, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:52.352869 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca837595-5507-4f9b-a31d-c7e38b27b19a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:52.367729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 952/1862] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:52 2022] PUT /identity/v3/projects/3199ddb07797466cb5d88adf25a33d2c/users/41ecac1fa93740c6bccc370105eb26b8/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:52.370953 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ca90fd-289f-49ef-be99-d6eada8251c9 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.370953 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ca90fd-289f-49ef-be99-d6eada8251c9 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.370953 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ca90fd-289f-49ef-be99-d6eada8251c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.377531 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 911/1863] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:52 2022] POST /identity/v3/projects => generated 391 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:52.378395 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1d0936-947a-41f7-87ea-0403bbf13e9b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:52.405567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-c1ca90fd-289f-49ef-be99-d6eada8251c9 None None] MFA Rules not processed for user `88c250889ad947658bb7bdd72316a75a`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:52.407080 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c1ca90fd-289f-49ef-be99-d6eada8251c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:52.460213 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1d0936-947a-41f7-87ea-0403bbf13e9b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:52.466579 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5c2349-348e-4eec-8d5c-c2c3c7892c33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:52.467243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5c2349-348e-4eec-8d5c-c2c3c7892c33 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.467531 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5c2349-348e-4eec-8d5c-c2c3c7892c33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.467818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5c2349-348e-4eec-8d5c-c2c3c7892c33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.468491 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5c2349-348e-4eec-8d5c-c2c3c7892c33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:52.470153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5c2349-348e-4eec-8d5c-c2c3c7892c33 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:52.485233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 912/1864] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:52 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:52.485233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5c2349-348e-4eec-8d5c-c2c3c7892c33 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:52.488316 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 953/1865] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:52 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:52.489666 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ca90fd-289f-49ef-be99-d6eada8251c9 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:52.499116 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:52.505145 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5c2349-348e-4eec-8d5c-c2c3c7892c33 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:52.506891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ca90fd-289f-49ef-be99-d6eada8251c9 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:52.511445 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584baf38-93bc-42db-94f1-003f0087f22f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:52.511783 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584baf38-93bc-42db-94f1-003f0087f22f admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.512167 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584baf38-93bc-42db-94f1-003f0087f22f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.512577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584baf38-93bc-42db-94f1-003f0087f22f admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.513185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee74514-4171-4376-a173-d0abb5eea444 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:52.513328 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584baf38-93bc-42db-94f1-003f0087f22f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:52.513862 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee74514-4171-4376-a173-d0abb5eea444 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.514528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee74514-4171-4376-a173-d0abb5eea444 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.515052 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee74514-4171-4376-a173-d0abb5eea444 admin admin] PATH_INFO: `/v3/projects/3199ddb07797466cb5d88adf25a33d2c/users/41ecac1fa93740c6bccc370105eb26b8/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.515313 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584baf38-93bc-42db-94f1-003f0087f22f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:52.517881 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee74514-4171-4376-a173-d0abb5eea444 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3199ddb07797466cb5d88adf25a33d2c, user_id=41ecac1fa93740c6bccc370105eb26b8, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:52.521141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee74514-4171-4376-a173-d0abb5eea444 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:52.534973 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 954/1866] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:52 2022] PUT /identity/v3/projects/3199ddb07797466cb5d88adf25a33d2c/users/41ecac1fa93740c6bccc370105eb26b8/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:52.535497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee74514-4171-4376-a173-d0abb5eea444 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:52.540489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4ee74514-4171-4376-a173-d0abb5eea444 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:52.542586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 913/1867] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:52 2022] POST /identity/v3/users => generated 296 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:52.545912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0ae4d2-a5af-4b99-b468-33dd55ae3731 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.546203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0ae4d2-a5af-4b99-b468-33dd55ae3731 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.546485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0ae4d2-a5af-4b99-b468-33dd55ae3731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.583069 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-df0ae4d2-a5af-4b99-b468-33dd55ae3731 None None] MFA Rules not processed for user `41ecac1fa93740c6bccc370105eb26b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:52.584304 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-df0ae4d2-a5af-4b99-b468-33dd55ae3731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:52.613698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee74514-4171-4376-a173-d0abb5eea444 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:52.618261 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0242b879-017a-450b-9cb1-336c1ad336c9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:52.618982 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0242b879-017a-450b-9cb1-336c1ad336c9 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.619368 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0242b879-017a-450b-9cb1-336c1ad336c9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.619746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0242b879-017a-450b-9cb1-336c1ad336c9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.621919 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-0242b879-017a-450b-9cb1-336c1ad336c9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:52.635588 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:52.680817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 914/1868] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:52 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:52.681479 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0ae4d2-a5af-4b99-b468-33dd55ae3731 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:52.709541 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0242b879-017a-450b-9cb1-336c1ad336c9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:52.711025 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0242b879-017a-450b-9cb1-336c1ad336c9 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:52.721856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 955/1869] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:52 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:52.723167 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0242b879-017a-450b-9cb1-336c1ad336c9 service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:52.727020 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-0242b879-017a-450b-9cb1-336c1ad336c9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:52.737926 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0242b879-017a-450b-9cb1-336c1ad336c9 service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:52.742082 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d07b84-7496-4e1c-88e9-4123d662dc6f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:52.742698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d07b84-7496-4e1c-88e9-4123d662dc6f service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.743013 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d07b84-7496-4e1c-88e9-4123d662dc6f service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.743288 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d07b84-7496-4e1c-88e9-4123d662dc6f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.754004 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d07b84-7496-4e1c-88e9-4123d662dc6f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:52.755798 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d07b84-7496-4e1c-88e9-4123d662dc6f service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:52.756470 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0ae4d2-a5af-4b99-b468-33dd55ae3731 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:52.764576 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a264ee73-747c-43af-b0b2-c54f5f90a3cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:52.765209 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a264ee73-747c-43af-b0b2-c54f5f90a3cb admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.765485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a264ee73-747c-43af-b0b2-c54f5f90a3cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.765754 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a264ee73-747c-43af-b0b2-c54f5f90a3cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.766399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a264ee73-747c-43af-b0b2-c54f5f90a3cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:52.768037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a264ee73-747c-43af-b0b2-c54f5f90a3cb admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:52.769840 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 956/1870] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:52 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:52.783477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 915/1871] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:52 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:52.785519 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d07b84-7496-4e1c-88e9-4123d662dc6f service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:52.800102 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d07b84-7496-4e1c-88e9-4123d662dc6f service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:52.805536 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0701833-1142-4e95-a011-6cfdf28ad1e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:52.806146 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0701833-1142-4e95-a011-6cfdf28ad1e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.806426 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0701833-1142-4e95-a011-6cfdf28ad1e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.806699 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0701833-1142-4e95-a011-6cfdf28ad1e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.807538 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0701833-1142-4e95-a011-6cfdf28ad1e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:52.809221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0701833-1142-4e95-a011-6cfdf28ad1e1 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:52.824293 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 957/1872] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:52 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:52.831581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a264ee73-747c-43af-b0b2-c54f5f90a3cb admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:52.842633 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a264ee73-747c-43af-b0b2-c54f5f90a3cb admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:52.850454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3373ba-9caa-46fb-9d0a-df6421591966 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:52.851357 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3373ba-9caa-46fb-9d0a-df6421591966 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.851635 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3373ba-9caa-46fb-9d0a-df6421591966 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.851902 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3373ba-9caa-46fb-9d0a-df6421591966 admin admin] PATH_INFO: `/v3/projects/04cbd2422036437d960b91015dafd7c6/users/b3ef68e862be48a5a3f8d8de685997b8/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.885327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3373ba-9caa-46fb-9d0a-df6421591966 admin admin] RBAC: Authorizing `identity:create_grant(project_id=04cbd2422036437d960b91015dafd7c6, user_id=b3ef68e862be48a5a3f8d8de685997b8, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:52.888379 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3373ba-9caa-46fb-9d0a-df6421591966 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:52.908934 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 916/1873] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:52 2022] PUT /identity/v3/projects/04cbd2422036437d960b91015dafd7c6/users/b3ef68e862be48a5a3f8d8de685997b8/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:52.920207 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c7bf7e-ef7b-47b0-8c4a-bf37e1f4a496 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:52.920497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c7bf7e-ef7b-47b0-8c4a-bf37e1f4a496 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:52.920776 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c7bf7e-ef7b-47b0-8c4a-bf37e1f4a496 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:52.961340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-11c7bf7e-ef7b-47b0-8c4a-bf37e1f4a496 None None] MFA Rules not processed for user `b3ef68e862be48a5a3f8d8de685997b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:52.961340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-11c7bf7e-ef7b-47b0-8c4a-bf37e1f4a496 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:53.063704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 958/1874] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:52 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:53.118278 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3373ba-9caa-46fb-9d0a-df6421591966 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:53.191107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3373ba-9caa-46fb-9d0a-df6421591966 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:53.195567 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da25203-a24b-4e04-b373-ad04847d0dec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:53.196194 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da25203-a24b-4e04-b373-ad04847d0dec service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:53.196474 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da25203-a24b-4e04-b373-ad04847d0dec service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:53.196774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da25203-a24b-4e04-b373-ad04847d0dec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:53.207238 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da25203-a24b-4e04-b373-ad04847d0dec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:53.208733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da25203-a24b-4e04-b373-ad04847d0dec service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:53.219494 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 917/1875] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:53 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:53.327525 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c7bf7e-ef7b-47b0-8c4a-bf37e1f4a496 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:53.337332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:53.397288 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c7bf7e-ef7b-47b0-8c4a-bf37e1f4a496 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:53.402206 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48cc7c5-5264-4d6b-bf48-c093c574adb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:53.402930 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48cc7c5-5264-4d6b-bf48-c093c574adb7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:53.403353 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48cc7c5-5264-4d6b-bf48-c093c574adb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:53.403633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48cc7c5-5264-4d6b-bf48-c093c574adb7 admin admin] PATH_INFO: `/v3/users/b42f4237a09245d3a3df94ff8ae02526` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:53.404826 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48cc7c5-5264-4d6b-bf48-c093c574adb7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b42f4237a09245d3a3df94ff8ae02526)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:53.406807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48cc7c5-5264-4d6b-bf48-c093c574adb7 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:53.475933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b48cc7c5-5264-4d6b-bf48-c093c574adb7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b42f4237a09245d3a3df94ff8ae02526'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:53.476948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b48cc7c5-5264-4d6b-bf48-c093c574adb7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b42f4237a09245d3a3df94ff8ae02526'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:53.489140 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b48cc7c5-5264-4d6b-bf48-c093c574adb7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b42f4237a09245d3a3df94ff8ae02526'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:53.514136 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b48cc7c5-5264-4d6b-bf48-c093c574adb7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b42f4237a09245d3a3df94ff8ae02526'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:53.522599 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b48cc7c5-5264-4d6b-bf48-c093c574adb7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b42f4237a09245d3a3df94ff8ae02526'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:53.535592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 959/1876] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:53 2022] DELETE /identity/v3/users/b42f4237a09245d3a3df94ff8ae02526 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:53.540157 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c698d5c-2143-48dd-a7a5-35e54d4a7669 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:53.540452 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c698d5c-2143-48dd-a7a5-35e54d4a7669 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:53.540736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c698d5c-2143-48dd-a7a5-35e54d4a7669 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:53.566768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-9c698d5c-2143-48dd-a7a5-35e54d4a7669 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:53.568464 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9c698d5c-2143-48dd-a7a5-35e54d4a7669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:53.621542 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:53.636591 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 918/1877] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:53.646328 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48cc7c5-5264-4d6b-bf48-c093c574adb7 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:53.647766 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b48cc7c5-5264-4d6b-bf48-c093c574adb7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:53.705939 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48cc7c5-5264-4d6b-bf48-c093c574adb7 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:53.711062 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be9c473-0612-4c6a-bc7e-0177fb8a68d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:53.712012 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be9c473-0612-4c6a-bc7e-0177fb8a68d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:53.712464 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be9c473-0612-4c6a-bc7e-0177fb8a68d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:53.712918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be9c473-0612-4c6a-bc7e-0177fb8a68d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:53.728861 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be9c473-0612-4c6a-bc7e-0177fb8a68d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:53.731479 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be9c473-0612-4c6a-bc7e-0177fb8a68d4 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:53.743338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 960/1878] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:54.102646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c698d5c-2143-48dd-a7a5-35e54d4a7669 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:54.104178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9c698d5c-2143-48dd-a7a5-35e54d4a7669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:54.117300 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c698d5c-2143-48dd-a7a5-35e54d4a7669 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:54.123164 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1d3c49-fe49-473d-a0f9-d2944f24888a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:54.123903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1d3c49-fe49-473d-a0f9-d2944f24888a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:54.124262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1d3c49-fe49-473d-a0f9-d2944f24888a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:54.124654 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1d3c49-fe49-473d-a0f9-d2944f24888a admin admin] PATH_INFO: `/v3/projects/5073a7a54615493b8587ad0062b8b7a6` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:54.125878 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1d3c49-fe49-473d-a0f9-d2944f24888a admin admin] RBAC: Authorizing `identity:delete_project(project_id=5073a7a54615493b8587ad0062b8b7a6)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:54.127946 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1d3c49-fe49-473d-a0f9-d2944f24888a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:54.198534 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ea1d3c49-fe49-473d-a0f9-d2944f24888a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5073a7a54615493b8587ad0062b8b7a6', 'request_id': 'req-ea1d3c49-fe49-473d-a0f9-d2944f24888a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:54.206569 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ea1d3c49-fe49-473d-a0f9-d2944f24888a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5073a7a54615493b8587ad0062b8b7a6', 'request_id': 'req-ea1d3c49-fe49-473d-a0f9-d2944f24888a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:54.213605 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ea1d3c49-fe49-473d-a0f9-d2944f24888a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5073a7a54615493b8587ad0062b8b7a6', 'request_id': 'req-ea1d3c49-fe49-473d-a0f9-d2944f24888a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:54.224879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ea1d3c49-fe49-473d-a0f9-d2944f24888a admin admin] The token cache is being invalidate because project 5073a7a54615493b8587ad0062b8b7a6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:54.225171 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ea1d3c49-fe49-473d-a0f9-d2944f24888a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:54.225851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ea1d3c49-fe49-473d-a0f9-d2944f24888a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:54.227401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 919/1879] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:54 2022] DELETE /identity/v3/projects/5073a7a54615493b8587ad0062b8b7a6 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:54.231869 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be9c473-0612-4c6a-bc7e-0177fb8a68d4 service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:54.241621 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be9c473-0612-4c6a-bc7e-0177fb8a68d4 service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:54.271156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:54.304691 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab2f8b7-2eaf-454b-97ab-ef7e6459538d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:54.305481 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab2f8b7-2eaf-454b-97ab-ef7e6459538d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:54.305758 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab2f8b7-2eaf-454b-97ab-ef7e6459538d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:54.306028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab2f8b7-2eaf-454b-97ab-ef7e6459538d admin admin] PATH_INFO: `/v3/users/41ecac1fa93740c6bccc370105eb26b8` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:54.307291 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab2f8b7-2eaf-454b-97ab-ef7e6459538d admin admin] RBAC: Authorizing `identity:delete_user(user_id=41ecac1fa93740c6bccc370105eb26b8)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:54.309389 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab2f8b7-2eaf-454b-97ab-ef7e6459538d admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:54.357167 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-2ab2f8b7-2eaf-454b-97ab-ef7e6459538d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '41ecac1fa93740c6bccc370105eb26b8'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:54.357753 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-2ab2f8b7-2eaf-454b-97ab-ef7e6459538d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '41ecac1fa93740c6bccc370105eb26b8'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:54.366365 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-2ab2f8b7-2eaf-454b-97ab-ef7e6459538d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '41ecac1fa93740c6bccc370105eb26b8'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:54.380732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-2ab2f8b7-2eaf-454b-97ab-ef7e6459538d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '41ecac1fa93740c6bccc370105eb26b8'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:54.388449 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-2ab2f8b7-2eaf-454b-97ab-ef7e6459538d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '41ecac1fa93740c6bccc370105eb26b8'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:54.394547 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 961/1880] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:58:54 2022] DELETE /identity/v3/users/41ecac1fa93740c6bccc370105eb26b8 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:54.678520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1d3c49-fe49-473d-a0f9-d2944f24888a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:54.680007 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ea1d3c49-fe49-473d-a0f9-d2944f24888a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:54.693943 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1d3c49-fe49-473d-a0f9-d2944f24888a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:54.754619 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11abc664-d6f5-4654-9790-a1277d160f81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:54.755392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11abc664-d6f5-4654-9790-a1277d160f81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:54.755675 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11abc664-d6f5-4654-9790-a1277d160f81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:54.755952 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11abc664-d6f5-4654-9790-a1277d160f81 admin admin] PATH_INFO: `/v3/projects/3199ddb07797466cb5d88adf25a33d2c` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:54.757196 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11abc664-d6f5-4654-9790-a1277d160f81 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3199ddb07797466cb5d88adf25a33d2c)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:54.759158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11abc664-d6f5-4654-9790-a1277d160f81 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:54.806939 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-11abc664-d6f5-4654-9790-a1277d160f81 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3199ddb07797466cb5d88adf25a33d2c', 'request_id': 'req-11abc664-d6f5-4654-9790-a1277d160f81'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:54.815182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-11abc664-d6f5-4654-9790-a1277d160f81 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3199ddb07797466cb5d88adf25a33d2c', 'request_id': 'req-11abc664-d6f5-4654-9790-a1277d160f81'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:54.821652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-11abc664-d6f5-4654-9790-a1277d160f81 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3199ddb07797466cb5d88adf25a33d2c', 'request_id': 'req-11abc664-d6f5-4654-9790-a1277d160f81'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:54.833181 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-11abc664-d6f5-4654-9790-a1277d160f81 admin admin] The token cache is being invalidate because project 3199ddb07797466cb5d88adf25a33d2c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:58:54.833472 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-11abc664-d6f5-4654-9790-a1277d160f81 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:54.834197 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-11abc664-d6f5-4654-9790-a1277d160f81 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:58:54.835840 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 920/1881] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:58:54 2022] DELETE /identity/v3/projects/3199ddb07797466cb5d88adf25a33d2c => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:54.841290 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbc1023-fe1b-49b4-b802-1562b08c5b5d None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:54.841579 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbc1023-fe1b-49b4-b802-1562b08c5b5d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:54.841856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbc1023-fe1b-49b4-b802-1562b08c5b5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:54.861710 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-bbbc1023-fe1b-49b4-b802-1562b08c5b5d None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:54.862943 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-bbbc1023-fe1b-49b4-b802-1562b08c5b5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:54.924764 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 962/1882] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:54.934530 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81291ea-29bc-49e2-b092-6ba609e6e70e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:54.934824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81291ea-29bc-49e2-b092-6ba609e6e70e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:54.935139 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81291ea-29bc-49e2-b092-6ba609e6e70e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:54.949695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:54.958264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c81291ea-29bc-49e2-b092-6ba609e6e70e None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:54.959523 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c81291ea-29bc-49e2-b092-6ba609e6e70e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:54.970341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 921/1883] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:58:54.974898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbc1023-fe1b-49b4-b802-1562b08c5b5d None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:54.985365 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbc1023-fe1b-49b4-b802-1562b08c5b5d None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:54.990222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161b400b-e760-4680-b47c-31edddad9ddf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:54.990845 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161b400b-e760-4680-b47c-31edddad9ddf admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:54.991162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161b400b-e760-4680-b47c-31edddad9ddf admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:54.991434 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161b400b-e760-4680-b47c-31edddad9ddf admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:54.992086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161b400b-e760-4680-b47c-31edddad9ddf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:58:54.992391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161b400b-e760-4680-b47c-31edddad9ddf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:54.994048 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161b400b-e760-4680-b47c-31edddad9ddf admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:55.008120 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 963/1884] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:58:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:58:55.013201 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81291ea-29bc-49e2-b092-6ba609e6e70e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:55.023353 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81291ea-29bc-49e2-b092-6ba609e6e70e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:55.029523 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6c39f1-1fc6-454d-b3e4-04ad56a28d53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:55.030370 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6c39f1-1fc6-454d-b3e4-04ad56a28d53 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:55.030662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6c39f1-1fc6-454d-b3e4-04ad56a28d53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:55.030967 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6c39f1-1fc6-454d-b3e4-04ad56a28d53 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:55.031665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6c39f1-1fc6-454d-b3e4-04ad56a28d53 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:55.033654 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6c39f1-1fc6-454d-b3e4-04ad56a28d53 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:55.064982 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 922/1885] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:55 2022] POST /identity/v3/projects => generated 383 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:55.069862 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161b400b-e760-4680-b47c-31edddad9ddf admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:55.080183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161b400b-e760-4680-b47c-31edddad9ddf admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:55.115221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:55.148940 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a9130d-059a-4c73-aad7-597ee8f153de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:55.149554 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a9130d-059a-4c73-aad7-597ee8f153de admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:55.149834 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a9130d-059a-4c73-aad7-597ee8f153de admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:55.150108 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a9130d-059a-4c73-aad7-597ee8f153de admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:55.150745 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a9130d-059a-4c73-aad7-597ee8f153de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:55.152915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a9130d-059a-4c73-aad7-597ee8f153de admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:55.178341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 964/1886] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:55 2022] POST /identity/v3/users => generated 292 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:55.179545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6c39f1-1fc6-454d-b3e4-04ad56a28d53 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:55.193776 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6c39f1-1fc6-454d-b3e4-04ad56a28d53 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:55.199481 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b864cf46-caf1-47dc-9ec2-030a73240529 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:55.200124 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b864cf46-caf1-47dc-9ec2-030a73240529 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:55.200407 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b864cf46-caf1-47dc-9ec2-030a73240529 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:55.200925 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b864cf46-caf1-47dc-9ec2-030a73240529 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:55.201572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b864cf46-caf1-47dc-9ec2-030a73240529 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:55.203334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b864cf46-caf1-47dc-9ec2-030a73240529 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:55.214323 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 923/1887] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:55 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 00:58:55.227849 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a9130d-059a-4c73-aad7-597ee8f153de admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:55.228865 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a9130d-059a-4c73-aad7-597ee8f153de admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:55.234208 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3efdc92-6112-489b-a4dc-67cf700ae21a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:55.235053 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3efdc92-6112-489b-a4dc-67cf700ae21a admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:55.235191 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3efdc92-6112-489b-a4dc-67cf700ae21a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:55.235463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3efdc92-6112-489b-a4dc-67cf700ae21a admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:55.236104 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3efdc92-6112-489b-a4dc-67cf700ae21a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:55.237714 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3efdc92-6112-489b-a4dc-67cf700ae21a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:55.249377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 965/1888] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:55 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:58:55.253716 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b864cf46-caf1-47dc-9ec2-030a73240529 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:55.263892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b864cf46-caf1-47dc-9ec2-030a73240529 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:55.268382 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43c368b-8d17-413c-9498-408fe375f678 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:55.269022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43c368b-8d17-413c-9498-408fe375f678 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:55.269305 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43c368b-8d17-413c-9498-408fe375f678 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:55.269573 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43c368b-8d17-413c-9498-408fe375f678 admin admin] PATH_INFO: `/v3/projects/9d77c179761547f38c5eea8ddd4ec597/users/f7ec322387094ed8906fa0fa64a59e7d/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:55.284194 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43c368b-8d17-413c-9498-408fe375f678 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d77c179761547f38c5eea8ddd4ec597, user_id=f7ec322387094ed8906fa0fa64a59e7d, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:55.286899 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43c368b-8d17-413c-9498-408fe375f678 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:55.301041 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 924/1889] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:55 2022] PUT /identity/v3/projects/9d77c179761547f38c5eea8ddd4ec597/users/f7ec322387094ed8906fa0fa64a59e7d/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:58:55.312044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf3ebd3-bdd4-4133-8bc7-3a92ebc1c345 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:55.312384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf3ebd3-bdd4-4133-8bc7-3a92ebc1c345 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:55.312716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf3ebd3-bdd4-4133-8bc7-3a92ebc1c345 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:55.343479 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-3bf3ebd3-bdd4-4133-8bc7-3a92ebc1c345 None None] MFA Rules not processed for user `f7ec322387094ed8906fa0fa64a59e7d`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:55.344709 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-3bf3ebd3-bdd4-4133-8bc7-3a92ebc1c345 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:55.411930 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 966/1890] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:55 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:58:55.416429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43c368b-8d17-413c-9498-408fe375f678 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:55.478576 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43c368b-8d17-413c-9498-408fe375f678 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:55.484042 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea630c5-5f12-49ee-84ee-bc0d97aa2300 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:55.484651 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea630c5-5f12-49ee-84ee-bc0d97aa2300 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:55.484932 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea630c5-5f12-49ee-84ee-bc0d97aa2300 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:55.485208 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea630c5-5f12-49ee-84ee-bc0d97aa2300 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:55.485839 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea630c5-5f12-49ee-84ee-bc0d97aa2300 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:55.487818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea630c5-5f12-49ee-84ee-bc0d97aa2300 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:55.516114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 925/1891] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:58:55 2022] POST /identity/v3/projects => generated 385 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:58:55.520741 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf3ebd3-bdd4-4133-8bc7-3a92ebc1c345 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:55.535863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf3ebd3-bdd4-4133-8bc7-3a92ebc1c345 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:55.597746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f193c92f-fc82-4875-ae0d-4050013cf0e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:55.598369 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f193c92f-fc82-4875-ae0d-4050013cf0e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:55.598649 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f193c92f-fc82-4875-ae0d-4050013cf0e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:55.598954 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f193c92f-fc82-4875-ae0d-4050013cf0e7 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:55.599601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f193c92f-fc82-4875-ae0d-4050013cf0e7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:55.601524 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f193c92f-fc82-4875-ae0d-4050013cf0e7 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:55.626896 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 967/1892] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:58:55 2022] POST /identity/v3/users => generated 299 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:58:55.631330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea630c5-5f12-49ee-84ee-bc0d97aa2300 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:55.641950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea630c5-5f12-49ee-84ee-bc0d97aa2300 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:55.646799 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66a0f8b-3861-4653-ac1b-4f8bed93f487 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:55.647518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66a0f8b-3861-4653-ac1b-4f8bed93f487 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:55.647738 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66a0f8b-3861-4653-ac1b-4f8bed93f487 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:55.648009 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66a0f8b-3861-4653-ac1b-4f8bed93f487 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:55.648684 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c66a0f8b-3861-4653-ac1b-4f8bed93f487 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:55.650296 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c66a0f8b-3861-4653-ac1b-4f8bed93f487 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:55.663127 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 926/1893] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:55 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 00:58:55.668123 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f193c92f-fc82-4875-ae0d-4050013cf0e7 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:55.678566 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f193c92f-fc82-4875-ae0d-4050013cf0e7 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:55.683919 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c286f6-c327-4c37-baf5-ff4c1e318752 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:55.684589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c286f6-c327-4c37-baf5-ff4c1e318752 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:55.684869 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c286f6-c327-4c37-baf5-ff4c1e318752 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:55.685145 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c286f6-c327-4c37-baf5-ff4c1e318752 admin admin] PATH_INFO: `/v3/projects/ad38d71b8bad4b6ab371191320ea8faf/users/3407aa30cb3649c4a46a674f263d3130/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:55.705973 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c286f6-c327-4c37-baf5-ff4c1e318752 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ad38d71b8bad4b6ab371191320ea8faf, user_id=3407aa30cb3649c4a46a674f263d3130, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:55.708709 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c286f6-c327-4c37-baf5-ff4c1e318752 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:55.722408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 968/1894] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:55 2022] PUT /identity/v3/projects/ad38d71b8bad4b6ab371191320ea8faf/users/3407aa30cb3649c4a46a674f263d3130/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:55.726349 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66a0f8b-3861-4653-ac1b-4f8bed93f487 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:55.736386 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66a0f8b-3861-4653-ac1b-4f8bed93f487 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:55.792165 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5051c82-80e5-419d-99dc-630294f09107 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:55.792768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5051c82-80e5-419d-99dc-630294f09107 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:55.793045 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5051c82-80e5-419d-99dc-630294f09107 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:55.793313 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5051c82-80e5-419d-99dc-630294f09107 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:55.793961 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5051c82-80e5-419d-99dc-630294f09107 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:55.795600 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5051c82-80e5-419d-99dc-630294f09107 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:55.805870 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 927/1895] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:58:55 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 00:58:55.810010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c286f6-c327-4c37-baf5-ff4c1e318752 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:55.821159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c286f6-c327-4c37-baf5-ff4c1e318752 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:55.826678 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb9be90-d7d7-4d53-9db0-89bcedd470b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:55.827374 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb9be90-d7d7-4d53-9db0-89bcedd470b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:55.827658 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb9be90-d7d7-4d53-9db0-89bcedd470b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:55.827938 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb9be90-d7d7-4d53-9db0-89bcedd470b1 admin admin] PATH_INFO: `/v3/projects/ad38d71b8bad4b6ab371191320ea8faf/users/3407aa30cb3649c4a46a674f263d3130/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:55.829878 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb9be90-d7d7-4d53-9db0-89bcedd470b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ad38d71b8bad4b6ab371191320ea8faf, user_id=3407aa30cb3649c4a46a674f263d3130, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:55.833817 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb9be90-d7d7-4d53-9db0-89bcedd470b1 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:55.841961 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:55.848698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 969/1896] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:58:55 2022] PUT /identity/v3/projects/ad38d71b8bad4b6ab371191320ea8faf/users/3407aa30cb3649c4a46a674f263d3130/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:58:55.859571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618f5732-600b-4708-813d-162d7a8ebc7d None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:55.860114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618f5732-600b-4708-813d-162d7a8ebc7d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:55.860414 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618f5732-600b-4708-813d-162d7a8ebc7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:55.885947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:58:55.889761 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-618f5732-600b-4708-813d-162d7a8ebc7d None None] MFA Rules not processed for user `3407aa30cb3649c4a46a674f263d3130`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:58:55.892124 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-618f5732-600b-4708-813d-162d7a8ebc7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:55.966953 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 928/1897] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:58:55 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:58:55.977687 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb9be90-d7d7-4d53-9db0-89bcedd470b1 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:55.980037 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-7eb9be90-d7d7-4d53-9db0-89bcedd470b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:56.043440 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb9be90-d7d7-4d53-9db0-89bcedd470b1 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:56.047450 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e51bcaa-8939-4d15-8f48-eea53353b4e4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:56.048067 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e51bcaa-8939-4d15-8f48-eea53353b4e4 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:56.048366 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e51bcaa-8939-4d15-8f48-eea53353b4e4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:56.048620 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e51bcaa-8939-4d15-8f48-eea53353b4e4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:56.106436 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e51bcaa-8939-4d15-8f48-eea53353b4e4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:56.107933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e51bcaa-8939-4d15-8f48-eea53353b4e4 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:56.117116 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 970/1898] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:55 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:56.612972 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618f5732-600b-4708-813d-162d7a8ebc7d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:56.624884 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618f5732-600b-4708-813d-162d7a8ebc7d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:56.629339 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ba04d5-f4e1-4b71-9672-ff45f2db4bc6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:56.629957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ba04d5-f4e1-4b71-9672-ff45f2db4bc6 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:56.630241 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ba04d5-f4e1-4b71-9672-ff45f2db4bc6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:56.630515 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ba04d5-f4e1-4b71-9672-ff45f2db4bc6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:56.632591 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d7ba04d5-f4e1-4b71-9672-ff45f2db4bc6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:56.718187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ba04d5-f4e1-4b71-9672-ff45f2db4bc6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:56.719787 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ba04d5-f4e1-4b71-9672-ff45f2db4bc6 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:56.731001 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 929/1899] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:56 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:58:57.390662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e51bcaa-8939-4d15-8f48-eea53353b4e4 service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:58:57.392260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4e51bcaa-8939-4d15-8f48-eea53353b4e4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:57.405175 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e51bcaa-8939-4d15-8f48-eea53353b4e4 service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:58:57.409136 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9095f8e4-60ba-421f-90ce-9afa078d2feb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:58:57.409760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9095f8e4-60ba-421f-90ce-9afa078d2feb service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:58:57.410044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9095f8e4-60ba-421f-90ce-9afa078d2feb service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:58:57.410320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9095f8e4-60ba-421f-90ce-9afa078d2feb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:58:57.412084 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-9095f8e4-60ba-421f-90ce-9afa078d2feb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:58:57.487454 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9095f8e4-60ba-421f-90ce-9afa078d2feb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:58:57.488817 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9095f8e4-60ba-421f-90ce-9afa078d2feb service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:58:57.498700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 971/1900] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:58:57 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:59:04.696433 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ba04d5-f4e1-4b71-9672-ff45f2db4bc6 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:04.707757 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ba04d5-f4e1-4b71-9672-ff45f2db4bc6 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:04.714815 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8576c7-e7c9-4d03-ac52-b033cced5189 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:04.716058 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8576c7-e7c9-4d03-ac52-b033cced5189 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:04.716436 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8576c7-e7c9-4d03-ac52-b033cced5189 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:04.716857 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8576c7-e7c9-4d03-ac52-b033cced5189 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:04.728440 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8576c7-e7c9-4d03-ac52-b033cced5189 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:04.730681 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8576c7-e7c9-4d03-ac52-b033cced5189 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:04.746902 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 930/1901] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:04 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:59:05.659732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9095f8e4-60ba-421f-90ce-9afa078d2feb service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:05.661197 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-9095f8e4-60ba-421f-90ce-9afa078d2feb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:05.711254 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:05.728552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9095f8e4-60ba-421f-90ce-9afa078d2feb service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:05.733300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81eab97a-61bb-4cf1-85e6-a2aa2ada6734 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:05.733944 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81eab97a-61bb-4cf1-85e6-a2aa2ada6734 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:05.734221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81eab97a-61bb-4cf1-85e6-a2aa2ada6734 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:05.734494 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81eab97a-61bb-4cf1-85e6-a2aa2ada6734 admin admin] PATH_INFO: `/v3/users/445cb44c97724c618ae9c636c074f0ee` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:05.735647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81eab97a-61bb-4cf1-85e6-a2aa2ada6734 admin admin] RBAC: Authorizing `identity:delete_user(user_id=445cb44c97724c618ae9c636c074f0ee)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:05.737577 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81eab97a-61bb-4cf1-85e6-a2aa2ada6734 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:05.787257 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-81eab97a-61bb-4cf1-85e6-a2aa2ada6734 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '445cb44c97724c618ae9c636c074f0ee'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:05.787928 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-81eab97a-61bb-4cf1-85e6-a2aa2ada6734 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '445cb44c97724c618ae9c636c074f0ee'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:05.795492 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-81eab97a-61bb-4cf1-85e6-a2aa2ada6734 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '445cb44c97724c618ae9c636c074f0ee'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:05.808846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-81eab97a-61bb-4cf1-85e6-a2aa2ada6734 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '445cb44c97724c618ae9c636c074f0ee'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:05.815887 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-81eab97a-61bb-4cf1-85e6-a2aa2ada6734 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '445cb44c97724c618ae9c636c074f0ee'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:05.821929 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 972/1902] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:59:05 2022] DELETE /identity/v3/users/445cb44c97724c618ae9c636c074f0ee => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:05.827450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6212fed-7273-4511-b4cc-cf9deb109561 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:05.828091 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6212fed-7273-4511-b4cc-cf9deb109561 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:05.828091 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6212fed-7273-4511-b4cc-cf9deb109561 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:05.854387 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d6212fed-7273-4511-b4cc-cf9deb109561 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:05.856514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d6212fed-7273-4511-b4cc-cf9deb109561 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:05.956560 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 931/1903] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:05.967019 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81eab97a-61bb-4cf1-85e6-a2aa2ada6734 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:05.968405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-81eab97a-61bb-4cf1-85e6-a2aa2ada6734 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:06.030821 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81eab97a-61bb-4cf1-85e6-a2aa2ada6734 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:06.034739 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffd82e1-3648-4652-a17b-4352d65ffb70 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:06.035477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffd82e1-3648-4652-a17b-4352d65ffb70 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:06.035817 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffd82e1-3648-4652-a17b-4352d65ffb70 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:06.036152 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffd82e1-3648-4652-a17b-4352d65ffb70 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:06.047407 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffd82e1-3648-4652-a17b-4352d65ffb70 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:06.048855 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffd82e1-3648-4652-a17b-4352d65ffb70 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:06.058950 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 973/1904] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:59:06.365026 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6212fed-7273-4511-b4cc-cf9deb109561 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:06.367224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d6212fed-7273-4511-b4cc-cf9deb109561 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:06.381861 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6212fed-7273-4511-b4cc-cf9deb109561 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:06.388348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4430c9b9-acd1-4944-aca5-4f105a512ae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:06.389096 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4430c9b9-acd1-4944-aca5-4f105a512ae2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:06.390269 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4430c9b9-acd1-4944-aca5-4f105a512ae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:06.390979 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4430c9b9-acd1-4944-aca5-4f105a512ae2 admin admin] PATH_INFO: `/v3/projects/5f810e546f6641cabdf5eb3ba3469c6b` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:06.392286 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4430c9b9-acd1-4944-aca5-4f105a512ae2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f810e546f6641cabdf5eb3ba3469c6b)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:06.394799 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4430c9b9-acd1-4944-aca5-4f105a512ae2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:06.419296 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:06.448458 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4430c9b9-acd1-4944-aca5-4f105a512ae2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f810e546f6641cabdf5eb3ba3469c6b', 'request_id': 'req-4430c9b9-acd1-4944-aca5-4f105a512ae2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:06.456512 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4430c9b9-acd1-4944-aca5-4f105a512ae2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f810e546f6641cabdf5eb3ba3469c6b', 'request_id': 'req-4430c9b9-acd1-4944-aca5-4f105a512ae2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:06.464659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4430c9b9-acd1-4944-aca5-4f105a512ae2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f810e546f6641cabdf5eb3ba3469c6b', 'request_id': 'req-4430c9b9-acd1-4944-aca5-4f105a512ae2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:06.475753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4430c9b9-acd1-4944-aca5-4f105a512ae2 admin admin] The token cache is being invalidate because project 5f810e546f6641cabdf5eb3ba3469c6b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:59:06.475950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4430c9b9-acd1-4944-aca5-4f105a512ae2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:06.476607 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4430c9b9-acd1-4944-aca5-4f105a512ae2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:06.478105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 932/1905] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:59:06 2022] DELETE /identity/v3/projects/5f810e546f6641cabdf5eb3ba3469c6b => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:06.482490 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffd82e1-3648-4652-a17b-4352d65ffb70 service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:06.492388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffd82e1-3648-4652-a17b-4352d65ffb70 service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:06.554616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a6c3bd-7437-4f75-9b5b-7bb76d5ac565 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:06.555330 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a6c3bd-7437-4f75-9b5b-7bb76d5ac565 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:06.555609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a6c3bd-7437-4f75-9b5b-7bb76d5ac565 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:06.555881 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a6c3bd-7437-4f75-9b5b-7bb76d5ac565 admin admin] PATH_INFO: `/v3/users/ee8c3d18b31646548bc4c82422bd6f4c` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:06.557519 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a6c3bd-7437-4f75-9b5b-7bb76d5ac565 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee8c3d18b31646548bc4c82422bd6f4c)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:06.559628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a6c3bd-7437-4f75-9b5b-7bb76d5ac565 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:06.570593 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:06.610590 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-75a6c3bd-7437-4f75-9b5b-7bb76d5ac565 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee8c3d18b31646548bc4c82422bd6f4c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:06.611274 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-75a6c3bd-7437-4f75-9b5b-7bb76d5ac565 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee8c3d18b31646548bc4c82422bd6f4c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:06.618879 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-75a6c3bd-7437-4f75-9b5b-7bb76d5ac565 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee8c3d18b31646548bc4c82422bd6f4c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:06.632711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-75a6c3bd-7437-4f75-9b5b-7bb76d5ac565 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee8c3d18b31646548bc4c82422bd6f4c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:06.640716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-75a6c3bd-7437-4f75-9b5b-7bb76d5ac565 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee8c3d18b31646548bc4c82422bd6f4c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:06.646993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 974/1906] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:59:06 2022] DELETE /identity/v3/users/ee8c3d18b31646548bc4c82422bd6f4c => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:06.937730 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4430c9b9-acd1-4944-aca5-4f105a512ae2 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:06.939328 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4430c9b9-acd1-4944-aca5-4f105a512ae2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:06.958200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4430c9b9-acd1-4944-aca5-4f105a512ae2 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:07.019132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b984481-a742-4b34-b889-8406c41b2313 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:07.019692 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b984481-a742-4b34-b889-8406c41b2313 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:07.019973 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b984481-a742-4b34-b889-8406c41b2313 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:07.020245 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b984481-a742-4b34-b889-8406c41b2313 admin admin] PATH_INFO: `/v3/projects/6d3280811ab24bc884551d5446b5787b` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:07.021374 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b984481-a742-4b34-b889-8406c41b2313 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d3280811ab24bc884551d5446b5787b)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:07.023350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b984481-a742-4b34-b889-8406c41b2313 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:07.070838 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4b984481-a742-4b34-b889-8406c41b2313 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d3280811ab24bc884551d5446b5787b', 'request_id': 'req-4b984481-a742-4b34-b889-8406c41b2313'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:07.080445 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4b984481-a742-4b34-b889-8406c41b2313 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d3280811ab24bc884551d5446b5787b', 'request_id': 'req-4b984481-a742-4b34-b889-8406c41b2313'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:07.087138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4b984481-a742-4b34-b889-8406c41b2313 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d3280811ab24bc884551d5446b5787b', 'request_id': 'req-4b984481-a742-4b34-b889-8406c41b2313'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:07.097551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4b984481-a742-4b34-b889-8406c41b2313 admin admin] The token cache is being invalidate because project 6d3280811ab24bc884551d5446b5787b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:59:07.097838 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4b984481-a742-4b34-b889-8406c41b2313 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:07.098477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4b984481-a742-4b34-b889-8406c41b2313 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:07.099989 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 933/1907] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:59:06 2022] DELETE /identity/v3/projects/6d3280811ab24bc884551d5446b5787b => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:07.104522 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a6c3bd-7437-4f75-9b5b-7bb76d5ac565 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:07.105972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-75a6c3bd-7437-4f75-9b5b-7bb76d5ac565 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:07.118731 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a6c3bd-7437-4f75-9b5b-7bb76d5ac565 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:07.183530 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed28839-b1e2-4ad6-87b0-8f5cac05e53d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:07.184434 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed28839-b1e2-4ad6-87b0-8f5cac05e53d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:07.184434 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed28839-b1e2-4ad6-87b0-8f5cac05e53d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:07.188823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed28839-b1e2-4ad6-87b0-8f5cac05e53d admin admin] PATH_INFO: `/v3/users/88c250889ad947658bb7bdd72316a75a` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:07.188823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed28839-b1e2-4ad6-87b0-8f5cac05e53d admin admin] RBAC: Authorizing `identity:delete_user(user_id=88c250889ad947658bb7bdd72316a75a)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:07.188823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed28839-b1e2-4ad6-87b0-8f5cac05e53d admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:07.242585 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-7ed28839-b1e2-4ad6-87b0-8f5cac05e53d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '88c250889ad947658bb7bdd72316a75a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:07.243381 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-7ed28839-b1e2-4ad6-87b0-8f5cac05e53d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '88c250889ad947658bb7bdd72316a75a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:07.252279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-7ed28839-b1e2-4ad6-87b0-8f5cac05e53d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '88c250889ad947658bb7bdd72316a75a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:07.272101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-7ed28839-b1e2-4ad6-87b0-8f5cac05e53d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '88c250889ad947658bb7bdd72316a75a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:07.289937 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-7ed28839-b1e2-4ad6-87b0-8f5cac05e53d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '88c250889ad947658bb7bdd72316a75a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:07.297865 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 975/1908] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:59:07 2022] DELETE /identity/v3/users/88c250889ad947658bb7bdd72316a75a => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:07.629506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b984481-a742-4b34-b889-8406c41b2313 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:07.631201 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4b984481-a742-4b34-b889-8406c41b2313 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:07.646303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b984481-a742-4b34-b889-8406c41b2313 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:07.682820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:07.712210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5644e305-23af-4031-9fa7-d8ac9f1f5761 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:07.712924 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5644e305-23af-4031-9fa7-d8ac9f1f5761 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:07.713752 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5644e305-23af-4031-9fa7-d8ac9f1f5761 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:07.714030 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5644e305-23af-4031-9fa7-d8ac9f1f5761 admin admin] PATH_INFO: `/v3/projects/dec347208ed246f9a7c569f90c46f9ac` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:07.715323 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5644e305-23af-4031-9fa7-d8ac9f1f5761 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dec347208ed246f9a7c569f90c46f9ac)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:07.717443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5644e305-23af-4031-9fa7-d8ac9f1f5761 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:07.767485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5644e305-23af-4031-9fa7-d8ac9f1f5761 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dec347208ed246f9a7c569f90c46f9ac', 'request_id': 'req-5644e305-23af-4031-9fa7-d8ac9f1f5761'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:07.776470 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5644e305-23af-4031-9fa7-d8ac9f1f5761 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dec347208ed246f9a7c569f90c46f9ac', 'request_id': 'req-5644e305-23af-4031-9fa7-d8ac9f1f5761'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:07.783752 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5644e305-23af-4031-9fa7-d8ac9f1f5761 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dec347208ed246f9a7c569f90c46f9ac', 'request_id': 'req-5644e305-23af-4031-9fa7-d8ac9f1f5761'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:07.795295 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5644e305-23af-4031-9fa7-d8ac9f1f5761 admin admin] The token cache is being invalidate because project dec347208ed246f9a7c569f90c46f9ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:59:07.795590 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5644e305-23af-4031-9fa7-d8ac9f1f5761 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:07.796275 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5644e305-23af-4031-9fa7-d8ac9f1f5761 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:07.797799 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 934/1909] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:59:07 2022] DELETE /identity/v3/projects/dec347208ed246f9a7c569f90c46f9ac => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:07.803799 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec8ffbe-ea85-4cf3-ab99-a8fe8e0c4e91 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:07.804026 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec8ffbe-ea85-4cf3-ab99-a8fe8e0c4e91 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:07.804305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec8ffbe-ea85-4cf3-ab99-a8fe8e0c4e91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:07.818236 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:DELETE=20,keystone:SELECT=78,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:07.826412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-8ec8ffbe-ea85-4cf3-ab99-a8fe8e0c4e91 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:07.828993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8ec8ffbe-ea85-4cf3-ab99-a8fe8e0c4e91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:07.895811 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 976/1910] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:07.905059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d9b524-7407-4fe9-aad5-f746c08eb9d0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:07.905345 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d9b524-7407-4fe9-aad5-f746c08eb9d0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:07.905628 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d9b524-7407-4fe9-aad5-f746c08eb9d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:07.928031 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e2d9b524-7407-4fe9-aad5-f746c08eb9d0 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:07.929278 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e2d9b524-7407-4fe9-aad5-f746c08eb9d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:07.938017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 935/1911] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:59:07.942409 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec8ffbe-ea85-4cf3-ab99-a8fe8e0c4e91 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:07.956629 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec8ffbe-ea85-4cf3-ab99-a8fe8e0c4e91 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:07.961826 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a749c80-6387-4334-a0a2-a5c687a12228 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:07.962465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a749c80-6387-4334-a0a2-a5c687a12228 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:07.962749 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a749c80-6387-4334-a0a2-a5c687a12228 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:07.963628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a749c80-6387-4334-a0a2-a5c687a12228 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:07.964680 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a749c80-6387-4334-a0a2-a5c687a12228 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:59:07.964988 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a749c80-6387-4334-a0a2-a5c687a12228 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:07.966673 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a749c80-6387-4334-a0a2-a5c687a12228 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:07.980990 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 977/1912] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:59:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:59:07.985549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d9b524-7407-4fe9-aad5-f746c08eb9d0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:07.995660 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d9b524-7407-4fe9-aad5-f746c08eb9d0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:08.000845 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cf356f-e8d6-4c29-bd78-294d530789f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:08.001583 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cf356f-e8d6-4c29-bd78-294d530789f2 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:08.001866 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cf356f-e8d6-4c29-bd78-294d530789f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:08.002191 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cf356f-e8d6-4c29-bd78-294d530789f2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:08.002809 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cf356f-e8d6-4c29-bd78-294d530789f2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:08.004952 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cf356f-e8d6-4c29-bd78-294d530789f2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:08.040519 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 936/1913] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:59:07 2022] POST /identity/v3/projects => generated 383 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:59:08.050421 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a749c80-6387-4334-a0a2-a5c687a12228 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:08.062491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a749c80-6387-4334-a0a2-a5c687a12228 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:08.139764 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed657824-b31c-4766-a396-876d6ddc43a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:08.140926 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed657824-b31c-4766-a396-876d6ddc43a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:08.140926 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed657824-b31c-4766-a396-876d6ddc43a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:08.140926 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed657824-b31c-4766-a396-876d6ddc43a2 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:08.141658 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed657824-b31c-4766-a396-876d6ddc43a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:08.143754 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed657824-b31c-4766-a396-876d6ddc43a2 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:08.168998 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 978/1914] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:59:08 2022] POST /identity/v3/users => generated 292 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:59:08.174117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cf356f-e8d6-4c29-bd78-294d530789f2 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:08.190336 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cf356f-e8d6-4c29-bd78-294d530789f2 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:08.195182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52e1165-c095-4914-8a8a-7859464a3353 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:08.195885 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52e1165-c095-4914-8a8a-7859464a3353 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:08.196284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52e1165-c095-4914-8a8a-7859464a3353 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:08.196553 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52e1165-c095-4914-8a8a-7859464a3353 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:08.197222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52e1165-c095-4914-8a8a-7859464a3353 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:08.199472 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52e1165-c095-4914-8a8a-7859464a3353 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:08.211744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 937/1915] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:59:08 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 00:59:08.217619 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed657824-b31c-4766-a396-876d6ddc43a2 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:08.230562 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed657824-b31c-4766-a396-876d6ddc43a2 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:08.235954 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2becaf45-ee7e-4d5c-b258-df6f2942d334 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:08.236578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2becaf45-ee7e-4d5c-b258-df6f2942d334 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:08.236858 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2becaf45-ee7e-4d5c-b258-df6f2942d334 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:08.237128 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2becaf45-ee7e-4d5c-b258-df6f2942d334 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:08.237770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2becaf45-ee7e-4d5c-b258-df6f2942d334 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:08.239422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2becaf45-ee7e-4d5c-b258-df6f2942d334 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:08.252010 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 979/1916] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:59:08 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:59:08.257254 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52e1165-c095-4914-8a8a-7859464a3353 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:08.276380 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52e1165-c095-4914-8a8a-7859464a3353 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:08.281591 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d84a6e-b5f8-4814-b134-8d748f9feed4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:08.282263 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d84a6e-b5f8-4814-b134-8d748f9feed4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:08.282545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d84a6e-b5f8-4814-b134-8d748f9feed4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:08.282826 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d84a6e-b5f8-4814-b134-8d748f9feed4 admin admin] PATH_INFO: `/v3/projects/ed6d1bcd141e4a46a88a759f744a4fe2/users/6c02e44b8b284b28b437c08ca1de1865/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:08.302284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d84a6e-b5f8-4814-b134-8d748f9feed4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ed6d1bcd141e4a46a88a759f744a4fe2, user_id=6c02e44b8b284b28b437c08ca1de1865, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:08.305056 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d84a6e-b5f8-4814-b134-8d748f9feed4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:08.319213 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 938/1917] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:59:08 2022] PUT /identity/v3/projects/ed6d1bcd141e4a46a88a759f744a4fe2/users/6c02e44b8b284b28b437c08ca1de1865/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:08.329906 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce0b5f6-60f3-49a9-b0c6-6a86e523412e None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:08.330201 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce0b5f6-60f3-49a9-b0c6-6a86e523412e None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:08.330481 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce0b5f6-60f3-49a9-b0c6-6a86e523412e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:08.363788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-6ce0b5f6-60f3-49a9-b0c6-6a86e523412e None None] MFA Rules not processed for user `6c02e44b8b284b28b437c08ca1de1865`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:08.364960 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6ce0b5f6-60f3-49a9-b0c6-6a86e523412e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:08.466443 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 980/1918] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:08 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:08.479452 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d84a6e-b5f8-4814-b134-8d748f9feed4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:08.484576 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d4d84a6e-b5f8-4814-b134-8d748f9feed4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:08.561425 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d84a6e-b5f8-4814-b134-8d748f9feed4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:08.565357 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e83b871-24ac-41be-ad98-90500421aeb9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:08.566034 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e83b871-24ac-41be-ad98-90500421aeb9 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:08.566308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e83b871-24ac-41be-ad98-90500421aeb9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:08.566581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e83b871-24ac-41be-ad98-90500421aeb9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:08.577884 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e83b871-24ac-41be-ad98-90500421aeb9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:08.579336 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e83b871-24ac-41be-ad98-90500421aeb9 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:08.591944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 939/1919] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:08 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:59:11.360917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce0b5f6-60f3-49a9-b0c6-6a86e523412e None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:11.365374 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:11.464007 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce0b5f6-60f3-49a9-b0c6-6a86e523412e None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:11.469473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b018c746-df03-4e85-9451-2566cda2d875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:11.470292 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b018c746-df03-4e85-9451-2566cda2d875 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:11.470558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b018c746-df03-4e85-9451-2566cda2d875 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:11.471089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b018c746-df03-4e85-9451-2566cda2d875 admin admin] PATH_INFO: `/v3/users/6c02e44b8b284b28b437c08ca1de1865` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:11.472305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b018c746-df03-4e85-9451-2566cda2d875 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6c02e44b8b284b28b437c08ca1de1865)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:11.474337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b018c746-df03-4e85-9451-2566cda2d875 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:11.529786 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b018c746-df03-4e85-9451-2566cda2d875 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6c02e44b8b284b28b437c08ca1de1865'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:11.530451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b018c746-df03-4e85-9451-2566cda2d875 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6c02e44b8b284b28b437c08ca1de1865'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:11.539742 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b018c746-df03-4e85-9451-2566cda2d875 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6c02e44b8b284b28b437c08ca1de1865'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:11.553176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b018c746-df03-4e85-9451-2566cda2d875 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6c02e44b8b284b28b437c08ca1de1865'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:11.560904 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b018c746-df03-4e85-9451-2566cda2d875 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6c02e44b8b284b28b437c08ca1de1865'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:11.567016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 981/1920] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:59:11 2022] DELETE /identity/v3/users/6c02e44b8b284b28b437c08ca1de1865 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:11.571522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a5282d-cabe-4d8b-a026-65d2893bb8b2 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:11.572065 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a5282d-cabe-4d8b-a026-65d2893bb8b2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:11.572189 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a5282d-cabe-4d8b-a026-65d2893bb8b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:11.585435 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:11.592296 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b1a5282d-cabe-4d8b-a026-65d2893bb8b2 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:11.594099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b1a5282d-cabe-4d8b-a026-65d2893bb8b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:11.653900 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 940/1921] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:59:11.663407 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b018c746-df03-4e85-9451-2566cda2d875 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:11.664805 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b018c746-df03-4e85-9451-2566cda2d875 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:11.728836 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b018c746-df03-4e85-9451-2566cda2d875 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:11.732520 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256f38dd-9f2c-42fd-bf3d-d6887f386cc8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:11.733142 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256f38dd-9f2c-42fd-bf3d-d6887f386cc8 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:11.733422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256f38dd-9f2c-42fd-bf3d-d6887f386cc8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:11.733697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256f38dd-9f2c-42fd-bf3d-d6887f386cc8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:11.743556 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256f38dd-9f2c-42fd-bf3d-d6887f386cc8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:11.745768 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256f38dd-9f2c-42fd-bf3d-d6887f386cc8 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:11.755797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 982/1922] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:59:12.065195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a5282d-cabe-4d8b-a026-65d2893bb8b2 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:12.066885 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b1a5282d-cabe-4d8b-a026-65d2893bb8b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:12.081032 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a5282d-cabe-4d8b-a026-65d2893bb8b2 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:12.088286 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65885701-715d-4449-8619-bdf40f4831d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:12.089149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65885701-715d-4449-8619-bdf40f4831d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:12.089991 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65885701-715d-4449-8619-bdf40f4831d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:12.090114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65885701-715d-4449-8619-bdf40f4831d8 admin admin] PATH_INFO: `/v3/projects/ed6d1bcd141e4a46a88a759f744a4fe2` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:12.091426 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65885701-715d-4449-8619-bdf40f4831d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ed6d1bcd141e4a46a88a759f744a4fe2)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:12.093586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65885701-715d-4449-8619-bdf40f4831d8 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:12.152246 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-65885701-715d-4449-8619-bdf40f4831d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ed6d1bcd141e4a46a88a759f744a4fe2', 'request_id': 'req-65885701-715d-4449-8619-bdf40f4831d8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:12.161467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-65885701-715d-4449-8619-bdf40f4831d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ed6d1bcd141e4a46a88a759f744a4fe2', 'request_id': 'req-65885701-715d-4449-8619-bdf40f4831d8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:12.172428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-65885701-715d-4449-8619-bdf40f4831d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ed6d1bcd141e4a46a88a759f744a4fe2', 'request_id': 'req-65885701-715d-4449-8619-bdf40f4831d8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:12.187107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-65885701-715d-4449-8619-bdf40f4831d8 admin admin] The token cache is being invalidate because project ed6d1bcd141e4a46a88a759f744a4fe2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:59:12.187564 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-65885701-715d-4449-8619-bdf40f4831d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:12.188250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-65885701-715d-4449-8619-bdf40f4831d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:12.190252 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 941/1923] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:59:12 2022] DELETE /identity/v3/projects/ed6d1bcd141e4a46a88a759f744a4fe2 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:12.195011 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab80451-e764-4f41-ae91-de3cc3b354f1 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:12.195303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab80451-e764-4f41-ae91-de3cc3b354f1 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:12.195579 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab80451-e764-4f41-ae91-de3cc3b354f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:12.219220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-dab80451-e764-4f41-ae91-de3cc3b354f1 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:12.220460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-dab80451-e764-4f41-ae91-de3cc3b354f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:12.241060 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:12.295013 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 983/1924] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:12.303504 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bd9e7b-c4f4-40fa-b8dc-7b98bde3c918 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:12.303821 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bd9e7b-c4f4-40fa-b8dc-7b98bde3c918 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:12.304098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bd9e7b-c4f4-40fa-b8dc-7b98bde3c918 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:12.326900 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-73bd9e7b-c4f4-40fa-b8dc-7b98bde3c918 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:12.328535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-73bd9e7b-c4f4-40fa-b8dc-7b98bde3c918 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:12.337391 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 942/1925] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:59:12.342883 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab80451-e764-4f41-ae91-de3cc3b354f1 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:12.355050 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab80451-e764-4f41-ae91-de3cc3b354f1 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:12.360958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d65d75d-ab72-47e5-ad38-2ce123fc1ac4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:12.361619 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d65d75d-ab72-47e5-ad38-2ce123fc1ac4 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:12.362156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d65d75d-ab72-47e5-ad38-2ce123fc1ac4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:12.362156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d65d75d-ab72-47e5-ad38-2ce123fc1ac4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:12.362840 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d65d75d-ab72-47e5-ad38-2ce123fc1ac4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:59:12.363165 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d65d75d-ab72-47e5-ad38-2ce123fc1ac4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:12.364883 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d65d75d-ab72-47e5-ad38-2ce123fc1ac4 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:12.378284 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 984/1926] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:59:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:59:12.383231 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bd9e7b-c4f4-40fa-b8dc-7b98bde3c918 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:12.393777 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bd9e7b-c4f4-40fa-b8dc-7b98bde3c918 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:12.399659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c391c3-13dc-419d-90f1-e676338a17ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:12.401177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c391c3-13dc-419d-90f1-e676338a17ee admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:12.401378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c391c3-13dc-419d-90f1-e676338a17ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:12.401656 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c391c3-13dc-419d-90f1-e676338a17ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:12.402306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c391c3-13dc-419d-90f1-e676338a17ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:12.404248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c391c3-13dc-419d-90f1-e676338a17ee admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:12.437059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 943/1927] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:59:12 2022] POST /identity/v3/projects => generated 373 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:59:12.441353 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d65d75d-ab72-47e5-ad38-2ce123fc1ac4 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:12.452717 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d65d75d-ab72-47e5-ad38-2ce123fc1ac4 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:12.519628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22feac2-b8f3-4763-9ee8-960d1a7ab97d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:12.520014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22feac2-b8f3-4763-9ee8-960d1a7ab97d admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:12.520543 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22feac2-b8f3-4763-9ee8-960d1a7ab97d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:12.520925 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22feac2-b8f3-4763-9ee8-960d1a7ab97d admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:12.521680 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22feac2-b8f3-4763-9ee8-960d1a7ab97d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:12.523860 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22feac2-b8f3-4763-9ee8-960d1a7ab97d admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:12.560942 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 985/1928] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:59:12 2022] POST /identity/v3/users => generated 287 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:59:12.565087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c391c3-13dc-419d-90f1-e676338a17ee admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:12.581680 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c391c3-13dc-419d-90f1-e676338a17ee admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:12.586597 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abf0b5d-2d41-4bbc-8592-0cb9602553ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:12.587265 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abf0b5d-2d41-4bbc-8592-0cb9602553ee admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:12.588039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abf0b5d-2d41-4bbc-8592-0cb9602553ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:12.588336 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abf0b5d-2d41-4bbc-8592-0cb9602553ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:12.589176 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abf0b5d-2d41-4bbc-8592-0cb9602553ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:12.591065 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abf0b5d-2d41-4bbc-8592-0cb9602553ee admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:12.606345 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 944/1929] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:59:12 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:59:12.607515 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22feac2-b8f3-4763-9ee8-960d1a7ab97d admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:12.618153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22feac2-b8f3-4763-9ee8-960d1a7ab97d admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:12.623924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce25698f-d2a1-4474-a6e7-059db810e0c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:12.624623 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce25698f-d2a1-4474-a6e7-059db810e0c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:12.624914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce25698f-d2a1-4474-a6e7-059db810e0c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:12.625191 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce25698f-d2a1-4474-a6e7-059db810e0c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:12.625844 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce25698f-d2a1-4474-a6e7-059db810e0c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:12.627510 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce25698f-d2a1-4474-a6e7-059db810e0c3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:12.639755 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 986/1930] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:59:12 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:59:12.644013 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abf0b5d-2d41-4bbc-8592-0cb9602553ee admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:12.654306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abf0b5d-2d41-4bbc-8592-0cb9602553ee admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:12.658969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d575e5f-0d66-43a7-b8b4-6fdb4b820f68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:12.659624 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d575e5f-0d66-43a7-b8b4-6fdb4b820f68 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:12.659905 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d575e5f-0d66-43a7-b8b4-6fdb4b820f68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:12.660183 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d575e5f-0d66-43a7-b8b4-6fdb4b820f68 admin admin] PATH_INFO: `/v3/projects/6569740dcccd4d4e979ea3c6f8d4b03a/users/93891bddcd6042d1885f5281a2c4d093/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:12.675176 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d575e5f-0d66-43a7-b8b4-6fdb4b820f68 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6569740dcccd4d4e979ea3c6f8d4b03a, user_id=93891bddcd6042d1885f5281a2c4d093, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:12.677847 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d575e5f-0d66-43a7-b8b4-6fdb4b820f68 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:12.691794 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 945/1931] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:59:12 2022] PUT /identity/v3/projects/6569740dcccd4d4e979ea3c6f8d4b03a/users/93891bddcd6042d1885f5281a2c4d093/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:12.703239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4f5a86-7832-45cb-9ae8-b53e1ae6f14c None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:12.703529 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4f5a86-7832-45cb-9ae8-b53e1ae6f14c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:12.703809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4f5a86-7832-45cb-9ae8-b53e1ae6f14c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:12.731482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-4f4f5a86-7832-45cb-9ae8-b53e1ae6f14c None None] MFA Rules not processed for user `93891bddcd6042d1885f5281a2c4d093`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:12.732704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4f4f5a86-7832-45cb-9ae8-b53e1ae6f14c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:12.800192 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 987/1932] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:12 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:12.814289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d575e5f-0d66-43a7-b8b4-6fdb4b820f68 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:12.816010 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4d575e5f-0d66-43a7-b8b4-6fdb4b820f68 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:12.848786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:12.878064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d575e5f-0d66-43a7-b8b4-6fdb4b820f68 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:12.881737 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3540dba0-b199-465e-bb93-d92f4e3aa865 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:12.882344 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3540dba0-b199-465e-bb93-d92f4e3aa865 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:12.882624 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3540dba0-b199-465e-bb93-d92f4e3aa865 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:12.882924 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3540dba0-b199-465e-bb93-d92f4e3aa865 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:12.891977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3540dba0-b199-465e-bb93-d92f4e3aa865 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:12.893932 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3540dba0-b199-465e-bb93-d92f4e3aa865 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:12.903462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 946/1933] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:12 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:59:12.929858 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4f5a86-7832-45cb-9ae8-b53e1ae6f14c None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:12.954271 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:13.002603 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4f5a86-7832-45cb-9ae8-b53e1ae6f14c None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:13.007299 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b946d5-50bb-4acd-8139-b822f54b7df3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:13.007947 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b946d5-50bb-4acd-8139-b822f54b7df3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:13.008230 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b946d5-50bb-4acd-8139-b822f54b7df3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:13.008506 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b946d5-50bb-4acd-8139-b822f54b7df3 admin admin] PATH_INFO: `/v3/users/93891bddcd6042d1885f5281a2c4d093` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:13.009713 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b946d5-50bb-4acd-8139-b822f54b7df3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=93891bddcd6042d1885f5281a2c4d093)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:13.012060 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b946d5-50bb-4acd-8139-b822f54b7df3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:13.067948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-97b946d5-50bb-4acd-8139-b822f54b7df3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '93891bddcd6042d1885f5281a2c4d093'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:13.068646 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-97b946d5-50bb-4acd-8139-b822f54b7df3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '93891bddcd6042d1885f5281a2c4d093'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:13.077531 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-97b946d5-50bb-4acd-8139-b822f54b7df3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '93891bddcd6042d1885f5281a2c4d093'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:13.094395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-97b946d5-50bb-4acd-8139-b822f54b7df3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '93891bddcd6042d1885f5281a2c4d093'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:13.101657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-97b946d5-50bb-4acd-8139-b822f54b7df3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '93891bddcd6042d1885f5281a2c4d093'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:13.108393 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 988/1934] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:59:12 2022] DELETE /identity/v3/users/93891bddcd6042d1885f5281a2c4d093 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:13.112929 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716520b6-0360-4738-91f6-f86c77b3b24e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:13.113417 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716520b6-0360-4738-91f6-f86c77b3b24e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:13.113697 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716520b6-0360-4738-91f6-f86c77b3b24e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:13.134440 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-716520b6-0360-4738-91f6-f86c77b3b24e None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:13.135794 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-716520b6-0360-4738-91f6-f86c77b3b24e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:13.204094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 947/1935] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:13.216058 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b946d5-50bb-4acd-8139-b822f54b7df3 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:13.217052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716520b6-0360-4738-91f6-f86c77b3b24e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:13.217652 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-97b946d5-50bb-4acd-8139-b822f54b7df3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:13.218457 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-716520b6-0360-4738-91f6-f86c77b3b24e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:13.231159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716520b6-0360-4738-91f6-f86c77b3b24e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:13.235856 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741d40ed-f41a-4d66-93a4-5ccf7f380c9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:13.236497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741d40ed-f41a-4d66-93a4-5ccf7f380c9b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:13.236795 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741d40ed-f41a-4d66-93a4-5ccf7f380c9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:13.237074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741d40ed-f41a-4d66-93a4-5ccf7f380c9b admin admin] PATH_INFO: `/v3/users/f7ec322387094ed8906fa0fa64a59e7d` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:13.238209 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741d40ed-f41a-4d66-93a4-5ccf7f380c9b admin admin] RBAC: Authorizing `identity:delete_user(user_id=f7ec322387094ed8906fa0fa64a59e7d)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:13.240318 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741d40ed-f41a-4d66-93a4-5ccf7f380c9b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:13.288771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b946d5-50bb-4acd-8139-b822f54b7df3 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:13.295274 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd7fb24-c452-4c8a-a8ca-fa402eb47a44 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:13.296143 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd7fb24-c452-4c8a-a8ca-fa402eb47a44 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:13.296556 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd7fb24-c452-4c8a-a8ca-fa402eb47a44 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:13.297339 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd7fb24-c452-4c8a-a8ca-fa402eb47a44 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:13.299014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-741d40ed-f41a-4d66-93a4-5ccf7f380c9b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f7ec322387094ed8906fa0fa64a59e7d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:13.299881 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-741d40ed-f41a-4d66-93a4-5ccf7f380c9b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f7ec322387094ed8906fa0fa64a59e7d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:13.308459 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-741d40ed-f41a-4d66-93a4-5ccf7f380c9b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f7ec322387094ed8906fa0fa64a59e7d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:13.309133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-741d40ed-f41a-4d66-93a4-5ccf7f380c9b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f7ec322387094ed8906fa0fa64a59e7d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:13.309722 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd7fb24-c452-4c8a-a8ca-fa402eb47a44 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:13.311295 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd7fb24-c452-4c8a-a8ca-fa402eb47a44 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:13.321751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 989/1936] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:59:13.324097 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-741d40ed-f41a-4d66-93a4-5ccf7f380c9b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f7ec322387094ed8906fa0fa64a59e7d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:13.338985 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 948/1937] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:59:13 2022] DELETE /identity/v3/users/f7ec322387094ed8906fa0fa64a59e7d => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:59:13.343517 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc22018-4ce5-4017-8395-7a7f901a5f79 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:13.343943 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc22018-4ce5-4017-8395-7a7f901a5f79 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:13.344443 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc22018-4ce5-4017-8395-7a7f901a5f79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:13.380183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-abc22018-4ce5-4017-8395-7a7f901a5f79 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:13.382169 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-abc22018-4ce5-4017-8395-7a7f901a5f79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:13.434200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:13.460140 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 990/1938] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:13.470649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741d40ed-f41a-4d66-93a4-5ccf7f380c9b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:13.472204 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-741d40ed-f41a-4d66-93a4-5ccf7f380c9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:13.531362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:13.551902 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741d40ed-f41a-4d66-93a4-5ccf7f380c9b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:13.555762 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93df1895-7c34-4de6-ab31-10e38244911a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:13.556470 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93df1895-7c34-4de6-ab31-10e38244911a service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:13.556927 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93df1895-7c34-4de6-ab31-10e38244911a service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:13.557261 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93df1895-7c34-4de6-ab31-10e38244911a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:13.569690 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93df1895-7c34-4de6-ab31-10e38244911a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:13.571204 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93df1895-7c34-4de6-ab31-10e38244911a service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:13.596076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 949/1939] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:59:13.673697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc22018-4ce5-4017-8395-7a7f901a5f79 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:13.675316 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-abc22018-4ce5-4017-8395-7a7f901a5f79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:13.688605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc22018-4ce5-4017-8395-7a7f901a5f79 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:13.694972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43d2f95-4e53-40c6-9d9f-2a833dab3600 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:13.695846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43d2f95-4e53-40c6-9d9f-2a833dab3600 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:13.696116 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43d2f95-4e53-40c6-9d9f-2a833dab3600 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:13.696496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43d2f95-4e53-40c6-9d9f-2a833dab3600 admin admin] PATH_INFO: `/v3/projects/6569740dcccd4d4e979ea3c6f8d4b03a` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:13.698629 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43d2f95-4e53-40c6-9d9f-2a833dab3600 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6569740dcccd4d4e979ea3c6f8d4b03a)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:13.701252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43d2f95-4e53-40c6-9d9f-2a833dab3600 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:13.792735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b43d2f95-4e53-40c6-9d9f-2a833dab3600 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6569740dcccd4d4e979ea3c6f8d4b03a', 'request_id': 'req-b43d2f95-4e53-40c6-9d9f-2a833dab3600'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:13.802154 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b43d2f95-4e53-40c6-9d9f-2a833dab3600 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6569740dcccd4d4e979ea3c6f8d4b03a', 'request_id': 'req-b43d2f95-4e53-40c6-9d9f-2a833dab3600'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:13.808637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b43d2f95-4e53-40c6-9d9f-2a833dab3600 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6569740dcccd4d4e979ea3c6f8d4b03a', 'request_id': 'req-b43d2f95-4e53-40c6-9d9f-2a833dab3600'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:13.820967 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b43d2f95-4e53-40c6-9d9f-2a833dab3600 admin admin] The token cache is being invalidate because project 6569740dcccd4d4e979ea3c6f8d4b03a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:59:13.821252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b43d2f95-4e53-40c6-9d9f-2a833dab3600 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:13.822460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b43d2f95-4e53-40c6-9d9f-2a833dab3600 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:13.824343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 991/1940] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:59:13 2022] DELETE /identity/v3/projects/6569740dcccd4d4e979ea3c6f8d4b03a => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:13.976007 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93df1895-7c34-4de6-ab31-10e38244911a service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:13.977551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-93df1895-7c34-4de6-ab31-10e38244911a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:13.998442 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93df1895-7c34-4de6-ab31-10e38244911a service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:14.066080 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af09999-8d65-4618-9a47-fba751df2b6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:14.066733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af09999-8d65-4618-9a47-fba751df2b6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:14.067056 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af09999-8d65-4618-9a47-fba751df2b6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:14.067335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af09999-8d65-4618-9a47-fba751df2b6a admin admin] PATH_INFO: `/v3/projects/9d77c179761547f38c5eea8ddd4ec597` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:14.068479 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af09999-8d65-4618-9a47-fba751df2b6a admin admin] RBAC: Authorizing `identity:delete_project(project_id=9d77c179761547f38c5eea8ddd4ec597)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:14.070415 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af09999-8d65-4618-9a47-fba751df2b6a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:14.122621 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8af09999-8d65-4618-9a47-fba751df2b6a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9d77c179761547f38c5eea8ddd4ec597', 'request_id': 'req-8af09999-8d65-4618-9a47-fba751df2b6a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:14.130283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8af09999-8d65-4618-9a47-fba751df2b6a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9d77c179761547f38c5eea8ddd4ec597', 'request_id': 'req-8af09999-8d65-4618-9a47-fba751df2b6a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:14.137392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8af09999-8d65-4618-9a47-fba751df2b6a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9d77c179761547f38c5eea8ddd4ec597', 'request_id': 'req-8af09999-8d65-4618-9a47-fba751df2b6a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:14.148877 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8af09999-8d65-4618-9a47-fba751df2b6a admin admin] The token cache is being invalidate because project 9d77c179761547f38c5eea8ddd4ec597 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:59:14.149220 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8af09999-8d65-4618-9a47-fba751df2b6a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:14.150441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8af09999-8d65-4618-9a47-fba751df2b6a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:14.152398 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 950/1941] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:59:13 2022] DELETE /identity/v3/projects/9d77c179761547f38c5eea8ddd4ec597 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:59:14.157203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43d2f95-4e53-40c6-9d9f-2a833dab3600 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:14.158912 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b43d2f95-4e53-40c6-9d9f-2a833dab3600 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:14.172408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43d2f95-4e53-40c6-9d9f-2a833dab3600 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:14.233765 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907464ac-43d0-4257-bdeb-36ac9482191b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:14.234332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907464ac-43d0-4257-bdeb-36ac9482191b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:14.234619 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907464ac-43d0-4257-bdeb-36ac9482191b admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:14.234938 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907464ac-43d0-4257-bdeb-36ac9482191b admin admin] PATH_INFO: `/v3/users/3407aa30cb3649c4a46a674f263d3130` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:14.236108 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907464ac-43d0-4257-bdeb-36ac9482191b admin admin] RBAC: Authorizing `identity:delete_user(user_id=3407aa30cb3649c4a46a674f263d3130)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:14.238066 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907464ac-43d0-4257-bdeb-36ac9482191b admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:14.285683 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-907464ac-43d0-4257-bdeb-36ac9482191b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3407aa30cb3649c4a46a674f263d3130'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:14.286375 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-907464ac-43d0-4257-bdeb-36ac9482191b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3407aa30cb3649c4a46a674f263d3130'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:14.295168 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-907464ac-43d0-4257-bdeb-36ac9482191b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3407aa30cb3649c4a46a674f263d3130'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:14.309004 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:14.312526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-907464ac-43d0-4257-bdeb-36ac9482191b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3407aa30cb3649c4a46a674f263d3130'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:14.323356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-907464ac-43d0-4257-bdeb-36ac9482191b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3407aa30cb3649c4a46a674f263d3130'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:14.329918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 992/1942] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:59:14 2022] DELETE /identity/v3/users/3407aa30cb3649c4a46a674f263d3130 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:59:14.629817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af09999-8d65-4618-9a47-fba751df2b6a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:14.631461 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8af09999-8d65-4618-9a47-fba751df2b6a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:14.647815 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af09999-8d65-4618-9a47-fba751df2b6a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:14.739375 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f885e64-99b7-4708-9f5a-eb0581c4f84d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:14.740272 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f885e64-99b7-4708-9f5a-eb0581c4f84d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:14.740670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f885e64-99b7-4708-9f5a-eb0581c4f84d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:14.741058 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f885e64-99b7-4708-9f5a-eb0581c4f84d admin admin] PATH_INFO: `/v3/projects/ad38d71b8bad4b6ab371191320ea8faf` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:14.742645 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f885e64-99b7-4708-9f5a-eb0581c4f84d admin admin] RBAC: Authorizing `identity:delete_project(project_id=ad38d71b8bad4b6ab371191320ea8faf)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:14.746607 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f885e64-99b7-4708-9f5a-eb0581c4f84d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:14.766356 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:14.805640 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0f885e64-99b7-4708-9f5a-eb0581c4f84d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ad38d71b8bad4b6ab371191320ea8faf', 'request_id': 'req-0f885e64-99b7-4708-9f5a-eb0581c4f84d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:14.815607 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0f885e64-99b7-4708-9f5a-eb0581c4f84d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ad38d71b8bad4b6ab371191320ea8faf', 'request_id': 'req-0f885e64-99b7-4708-9f5a-eb0581c4f84d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:14.824005 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0f885e64-99b7-4708-9f5a-eb0581c4f84d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ad38d71b8bad4b6ab371191320ea8faf', 'request_id': 'req-0f885e64-99b7-4708-9f5a-eb0581c4f84d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:14.827972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907464ac-43d0-4257-bdeb-36ac9482191b admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:14.829551 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-907464ac-43d0-4257-bdeb-36ac9482191b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:14.846294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907464ac-43d0-4257-bdeb-36ac9482191b admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:14.848730 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0f885e64-99b7-4708-9f5a-eb0581c4f84d admin admin] The token cache is being invalidate because project ad38d71b8bad4b6ab371191320ea8faf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:59:14.849269 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0f885e64-99b7-4708-9f5a-eb0581c4f84d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:14.850303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0f885e64-99b7-4708-9f5a-eb0581c4f84d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:14.852753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 951/1943] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:59:14 2022] DELETE /identity/v3/projects/ad38d71b8bad4b6ab371191320ea8faf => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:14.859136 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae2f0c0-fb58-4cdf-9854-c88f587d8a55 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:14.859652 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae2f0c0-fb58-4cdf-9854-c88f587d8a55 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:14.859934 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae2f0c0-fb58-4cdf-9854-c88f587d8a55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:14.883072 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-cae2f0c0-fb58-4cdf-9854-c88f587d8a55 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:14.884187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cae2f0c0-fb58-4cdf-9854-c88f587d8a55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:14.917777 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fe616f-ad17-43a2-97bb-7dcb36535b21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:14.918434 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fe616f-ad17-43a2-97bb-7dcb36535b21 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:14.918732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fe616f-ad17-43a2-97bb-7dcb36535b21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:14.919030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fe616f-ad17-43a2-97bb-7dcb36535b21 admin admin] PATH_INFO: `/v3/users/b3ef68e862be48a5a3f8d8de685997b8` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:14.920180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20fe616f-ad17-43a2-97bb-7dcb36535b21 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b3ef68e862be48a5a3f8d8de685997b8)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:14.922135 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20fe616f-ad17-43a2-97bb-7dcb36535b21 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:14.954156 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 952/1944] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:14.963608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f700a296-d238-4903-abb9-dc4468ca26ef None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:14.963988 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f700a296-d238-4903-abb9-dc4468ca26ef None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:14.964157 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f700a296-d238-4903-abb9-dc4468ca26ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:14.980746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-20fe616f-ad17-43a2-97bb-7dcb36535b21 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b3ef68e862be48a5a3f8d8de685997b8'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:14.981544 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-20fe616f-ad17-43a2-97bb-7dcb36535b21 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b3ef68e862be48a5a3f8d8de685997b8'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:14.985238 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f700a296-d238-4903-abb9-dc4468ca26ef None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:14.986546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f700a296-d238-4903-abb9-dc4468ca26ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:14.990033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-20fe616f-ad17-43a2-97bb-7dcb36535b21 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b3ef68e862be48a5a3f8d8de685997b8'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:15.004380 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 953/1945] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:59:15.008275 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f700a296-d238-4903-abb9-dc4468ca26ef None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:15.011880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-20fe616f-ad17-43a2-97bb-7dcb36535b21 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b3ef68e862be48a5a3f8d8de685997b8'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:15.018673 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f700a296-d238-4903-abb9-dc4468ca26ef None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:15.021793 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-20fe616f-ad17-43a2-97bb-7dcb36535b21 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b3ef68e862be48a5a3f8d8de685997b8'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:15.023845 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b7d6cf-255e-417b-b52c-26845a2fed43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:15.024482 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b7d6cf-255e-417b-b52c-26845a2fed43 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:15.024763 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b7d6cf-255e-417b-b52c-26845a2fed43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:15.025618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b7d6cf-255e-417b-b52c-26845a2fed43 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:15.026355 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b7d6cf-255e-417b-b52c-26845a2fed43 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:59:15.026691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b7d6cf-255e-417b-b52c-26845a2fed43 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:15.028421 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b7d6cf-255e-417b-b52c-26845a2fed43 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:15.040115 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 993/1946] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:59:14 2022] DELETE /identity/v3/users/b3ef68e862be48a5a3f8d8de685997b8 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:59:15.045349 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c301e55a-83f6-40fb-bb17-edee98f69727 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:15.048155 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c301e55a-83f6-40fb-bb17-edee98f69727 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:15.048353 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 954/1947] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:59:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:59:15.048472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c301e55a-83f6-40fb-bb17-edee98f69727 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:15.050004 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b7d6cf-255e-417b-b52c-26845a2fed43 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:15.051983 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-06b7d6cf-255e-417b-b52c-26845a2fed43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:15.066315 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-c301e55a-83f6-40fb-bb17-edee98f69727 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:15.067567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c301e55a-83f6-40fb-bb17-edee98f69727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:15.068162 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b7d6cf-255e-417b-b52c-26845a2fed43 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:15.135940 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015aefd3-2abe-4e35-8404-5e09cd279a4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:15.136849 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015aefd3-2abe-4e35-8404-5e09cd279a4f admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:15.137251 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015aefd3-2abe-4e35-8404-5e09cd279a4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:15.137569 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015aefd3-2abe-4e35-8404-5e09cd279a4f admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:15.138253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015aefd3-2abe-4e35-8404-5e09cd279a4f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:15.141234 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015aefd3-2abe-4e35-8404-5e09cd279a4f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:15.142563 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 994/1948] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:59:15.153327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c301e55a-83f6-40fb-bb17-edee98f69727 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:15.154792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c301e55a-83f6-40fb-bb17-edee98f69727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:15.166566 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:15.182703 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 955/1949] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:59:15 2022] POST /identity/v3/projects => generated 385 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:59:15.187977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015aefd3-2abe-4e35-8404-5e09cd279a4f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:15.205326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:15.237262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c301e55a-83f6-40fb-bb17-edee98f69727 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:15.265276 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015aefd3-2abe-4e35-8404-5e09cd279a4f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:15.270122 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a274c9b-8e78-48a4-8592-f88cff6d0ed8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:15.270737 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a274c9b-8e78-48a4-8592-f88cff6d0ed8 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:15.271048 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a274c9b-8e78-48a4-8592-f88cff6d0ed8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:15.271494 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a274c9b-8e78-48a4-8592-f88cff6d0ed8 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:15.272253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a274c9b-8e78-48a4-8592-f88cff6d0ed8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:15.274207 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a274c9b-8e78-48a4-8592-f88cff6d0ed8 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:15.290932 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ff0938-c936-4260-a077-a8a0da294933 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:15.291543 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ff0938-c936-4260-a077-a8a0da294933 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:15.291774 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ff0938-c936-4260-a077-a8a0da294933 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:15.292072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ff0938-c936-4260-a077-a8a0da294933 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:15.299510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 956/1950] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:59:15 2022] POST /identity/v3/users => generated 293 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:59:15.301971 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ff0938-c936-4260-a077-a8a0da294933 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:15.303499 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ff0938-c936-4260-a077-a8a0da294933 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:15.305105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a274c9b-8e78-48a4-8592-f88cff6d0ed8 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:15.314420 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 995/1951] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:59:15.316600 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a274c9b-8e78-48a4-8592-f88cff6d0ed8 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:15.330805 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36788fe6-0647-49f0-a046-7a7b00a8a16a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:15.330805 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36788fe6-0647-49f0-a046-7a7b00a8a16a admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:15.330805 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36788fe6-0647-49f0-a046-7a7b00a8a16a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:15.330805 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36788fe6-0647-49f0-a046-7a7b00a8a16a admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:15.331434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36788fe6-0647-49f0-a046-7a7b00a8a16a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:15.331434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36788fe6-0647-49f0-a046-7a7b00a8a16a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:15.342634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 957/1952] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:59:15 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:59:15.347457 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ff0938-c936-4260-a077-a8a0da294933 service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:15.359028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ff0938-c936-4260-a077-a8a0da294933 service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:15.364408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bca9f29-e184-4c06-8fe1-4c55ef84bfec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:15.365076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bca9f29-e184-4c06-8fe1-4c55ef84bfec admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:15.365319 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bca9f29-e184-4c06-8fe1-4c55ef84bfec admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:15.365644 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bca9f29-e184-4c06-8fe1-4c55ef84bfec admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:15.366668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bca9f29-e184-4c06-8fe1-4c55ef84bfec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:15.368523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bca9f29-e184-4c06-8fe1-4c55ef84bfec admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:15.381975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 996/1953] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:59:15 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:59:15.388209 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36788fe6-0647-49f0-a046-7a7b00a8a16a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:15.403561 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36788fe6-0647-49f0-a046-7a7b00a8a16a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:15.409121 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97061b15-8387-4980-b7be-490c2568e229 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:15.409890 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97061b15-8387-4980-b7be-490c2568e229 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:15.410702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97061b15-8387-4980-b7be-490c2568e229 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:15.410702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97061b15-8387-4980-b7be-490c2568e229 admin admin] PATH_INFO: `/v3/projects/f9e95af24f5949d5ace05479c100dcd2/users/9b1d3bf7c7684cf3812f51af101cffce/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:15.442098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97061b15-8387-4980-b7be-490c2568e229 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f9e95af24f5949d5ace05479c100dcd2, user_id=9b1d3bf7c7684cf3812f51af101cffce, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:15.444948 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97061b15-8387-4980-b7be-490c2568e229 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:15.461050 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 958/1954] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:59:15 2022] PUT /identity/v3/projects/f9e95af24f5949d5ace05479c100dcd2/users/9b1d3bf7c7684cf3812f51af101cffce/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:15.473664 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ed69ac-724b-4237-9555-ffd4611ce5e2 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:15.474160 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ed69ac-724b-4237-9555-ffd4611ce5e2 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:15.474160 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ed69ac-724b-4237-9555-ffd4611ce5e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:15.537658 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-01ed69ac-724b-4237-9555-ffd4611ce5e2 None None] MFA Rules not processed for user `9b1d3bf7c7684cf3812f51af101cffce`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:15.539028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-01ed69ac-724b-4237-9555-ffd4611ce5e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:15.631899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 997/1955] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:15 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:15.646423 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97061b15-8387-4980-b7be-490c2568e229 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:15.649130 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-97061b15-8387-4980-b7be-490c2568e229 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:15.675405 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ed69ac-724b-4237-9555-ffd4611ce5e2 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:15.677703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-01ed69ac-724b-4237-9555-ffd4611ce5e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:15.722321 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97061b15-8387-4980-b7be-490c2568e229 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:15.726732 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a71bc07-e2ca-468c-807b-84a7b1305cd3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:15.727763 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a71bc07-e2ca-468c-807b-84a7b1305cd3 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:15.728330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a71bc07-e2ca-468c-807b-84a7b1305cd3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:15.728612 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a71bc07-e2ca-468c-807b-84a7b1305cd3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:15.737725 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:15.738155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a71bc07-e2ca-468c-807b-84a7b1305cd3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:15.740100 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a71bc07-e2ca-468c-807b-84a7b1305cd3 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:15.752359 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 959/1956] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:15 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:59:15.763337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ed69ac-724b-4237-9555-ffd4611ce5e2 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:15.768065 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9126d6e-a2d1-4776-94a4-c3e8952d4677 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:15.768735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9126d6e-a2d1-4776-94a4-c3e8952d4677 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:15.769203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9126d6e-a2d1-4776-94a4-c3e8952d4677 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:15.769430 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9126d6e-a2d1-4776-94a4-c3e8952d4677 admin admin] PATH_INFO: `/v3/projects/04cbd2422036437d960b91015dafd7c6` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:15.770557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9126d6e-a2d1-4776-94a4-c3e8952d4677 admin admin] RBAC: Authorizing `identity:delete_project(project_id=04cbd2422036437d960b91015dafd7c6)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:15.772519 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9126d6e-a2d1-4776-94a4-c3e8952d4677 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:15.827751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b9126d6e-a2d1-4776-94a4-c3e8952d4677 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '04cbd2422036437d960b91015dafd7c6', 'request_id': 'req-b9126d6e-a2d1-4776-94a4-c3e8952d4677'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:15.836080 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b9126d6e-a2d1-4776-94a4-c3e8952d4677 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '04cbd2422036437d960b91015dafd7c6', 'request_id': 'req-b9126d6e-a2d1-4776-94a4-c3e8952d4677'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:15.843915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b9126d6e-a2d1-4776-94a4-c3e8952d4677 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '04cbd2422036437d960b91015dafd7c6', 'request_id': 'req-b9126d6e-a2d1-4776-94a4-c3e8952d4677'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:15.857994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b9126d6e-a2d1-4776-94a4-c3e8952d4677 admin admin] The token cache is being invalidate because project 04cbd2422036437d960b91015dafd7c6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:59:15.858341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b9126d6e-a2d1-4776-94a4-c3e8952d4677 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:15.859065 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b9126d6e-a2d1-4776-94a4-c3e8952d4677 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:15.860630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 998/1957] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:59:15 2022] DELETE /identity/v3/projects/04cbd2422036437d960b91015dafd7c6 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:25.747135 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=65,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:25.854893 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=32,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:27.475778 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a71bc07-e2ca-468c-807b-84a7b1305cd3 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:27.487813 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a71bc07-e2ca-468c-807b-84a7b1305cd3 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:27.544945 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1413f081-814e-49ca-bf86-f063f8767235 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:27.545581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1413f081-814e-49ca-bf86-f063f8767235 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:27.545857 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1413f081-814e-49ca-bf86-f063f8767235 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:27.546129 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1413f081-814e-49ca-bf86-f063f8767235 admin admin] PATH_INFO: `/v3/users/9b1d3bf7c7684cf3812f51af101cffce` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:27.547282 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1413f081-814e-49ca-bf86-f063f8767235 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9b1d3bf7c7684cf3812f51af101cffce)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:27.549212 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1413f081-814e-49ca-bf86-f063f8767235 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:27.601979 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1413f081-814e-49ca-bf86-f063f8767235 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9b1d3bf7c7684cf3812f51af101cffce'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:27.602562 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1413f081-814e-49ca-bf86-f063f8767235 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9b1d3bf7c7684cf3812f51af101cffce'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:27.611052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1413f081-814e-49ca-bf86-f063f8767235 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9b1d3bf7c7684cf3812f51af101cffce'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:27.611675 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1413f081-814e-49ca-bf86-f063f8767235 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9b1d3bf7c7684cf3812f51af101cffce'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:27.625253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1413f081-814e-49ca-bf86-f063f8767235 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9b1d3bf7c7684cf3812f51af101cffce'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:27.638684 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 960/1958] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:59:27 2022] DELETE /identity/v3/users/9b1d3bf7c7684cf3812f51af101cffce => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:27.643284 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de299b4-9ed3-4f2e-8b3f-821a9a806f6b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:27.644003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de299b4-9ed3-4f2e-8b3f-821a9a806f6b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:27.644003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de299b4-9ed3-4f2e-8b3f-821a9a806f6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:27.664525 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-2de299b4-9ed3-4f2e-8b3f-821a9a806f6b None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:27.666271 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-2de299b4-9ed3-4f2e-8b3f-821a9a806f6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:27.733202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 999/1959] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:27.744895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1413f081-814e-49ca-bf86-f063f8767235 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:27.746375 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1413f081-814e-49ca-bf86-f063f8767235 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:27.810445 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1413f081-814e-49ca-bf86-f063f8767235 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:27.814502 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8285fc-082c-4f1c-861f-fa6f6da5e871 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:27.815300 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8285fc-082c-4f1c-861f-fa6f6da5e871 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:27.815487 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8285fc-082c-4f1c-861f-fa6f6da5e871 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:27.815728 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8285fc-082c-4f1c-861f-fa6f6da5e871 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:27.827484 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8285fc-082c-4f1c-861f-fa6f6da5e871 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:27.828953 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8285fc-082c-4f1c-861f-fa6f6da5e871 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:27.839791 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 961/1960] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:59:28.165848 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de299b4-9ed3-4f2e-8b3f-821a9a806f6b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:28.165848 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-2de299b4-9ed3-4f2e-8b3f-821a9a806f6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:28.179708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de299b4-9ed3-4f2e-8b3f-821a9a806f6b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:28.186851 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990ddd32-3a22-475f-9a8d-4bf0fcccb1a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:28.187765 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990ddd32-3a22-475f-9a8d-4bf0fcccb1a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:28.187920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990ddd32-3a22-475f-9a8d-4bf0fcccb1a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:28.188189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990ddd32-3a22-475f-9a8d-4bf0fcccb1a3 admin admin] PATH_INFO: `/v3/projects/f9e95af24f5949d5ace05479c100dcd2` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:28.189358 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990ddd32-3a22-475f-9a8d-4bf0fcccb1a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f9e95af24f5949d5ace05479c100dcd2)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:28.191366 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990ddd32-3a22-475f-9a8d-4bf0fcccb1a3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:28.251243 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-990ddd32-3a22-475f-9a8d-4bf0fcccb1a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f9e95af24f5949d5ace05479c100dcd2', 'request_id': 'req-990ddd32-3a22-475f-9a8d-4bf0fcccb1a3'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:28.260696 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-990ddd32-3a22-475f-9a8d-4bf0fcccb1a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f9e95af24f5949d5ace05479c100dcd2', 'request_id': 'req-990ddd32-3a22-475f-9a8d-4bf0fcccb1a3'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:28.267699 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-990ddd32-3a22-475f-9a8d-4bf0fcccb1a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f9e95af24f5949d5ace05479c100dcd2', 'request_id': 'req-990ddd32-3a22-475f-9a8d-4bf0fcccb1a3'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:28.282222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-990ddd32-3a22-475f-9a8d-4bf0fcccb1a3 admin admin] The token cache is being invalidate because project f9e95af24f5949d5ace05479c100dcd2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:59:28.282538 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-990ddd32-3a22-475f-9a8d-4bf0fcccb1a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:28.283243 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-990ddd32-3a22-475f-9a8d-4bf0fcccb1a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:28.285474 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1000/1961] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:59:28 2022] DELETE /identity/v3/projects/f9e95af24f5949d5ace05479c100dcd2 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:28.291714 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90b10ec-2475-4ab6-9dc4-7562e13dfe42 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:28.292019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90b10ec-2475-4ab6-9dc4-7562e13dfe42 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:28.292294 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90b10ec-2475-4ab6-9dc4-7562e13dfe42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:28.313497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b90b10ec-2475-4ab6-9dc4-7562e13dfe42 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:28.315055 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b90b10ec-2475-4ab6-9dc4-7562e13dfe42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:28.347558 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:28.430904 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 962/1962] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 00:59:28.435615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dacb44e-6d9f-4888-9528-275e5a055a90 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:28.436241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dacb44e-6d9f-4888-9528-275e5a055a90 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:28.437332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dacb44e-6d9f-4888-9528-275e5a055a90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:28.468605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-0dacb44e-6d9f-4888-9528-275e5a055a90 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:28.470350 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-0dacb44e-6d9f-4888-9528-275e5a055a90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:28.480155 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1001/1963] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:28.484591 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90b10ec-2475-4ab6-9dc4-7562e13dfe42 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:28.495493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90b10ec-2475-4ab6-9dc4-7562e13dfe42 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:28.500269 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5208fae-8ced-494a-98bb-e661d580aa25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:28.500893 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5208fae-8ced-494a-98bb-e661d580aa25 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:28.501174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5208fae-8ced-494a-98bb-e661d580aa25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:28.501469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5208fae-8ced-494a-98bb-e661d580aa25 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:28.502118 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5208fae-8ced-494a-98bb-e661d580aa25 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:59:28.502423 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5208fae-8ced-494a-98bb-e661d580aa25 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:28.504252 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5208fae-8ced-494a-98bb-e661d580aa25 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:28.516910 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 963/1964] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:59:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:59:28.522324 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dacb44e-6d9f-4888-9528-275e5a055a90 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:28.532092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dacb44e-6d9f-4888-9528-275e5a055a90 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:28.537068 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb4dde9-dc34-4105-a8a1-5c04dfb3fda1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:28.537692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb4dde9-dc34-4105-a8a1-5c04dfb3fda1 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:28.537973 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb4dde9-dc34-4105-a8a1-5c04dfb3fda1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:28.538244 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb4dde9-dc34-4105-a8a1-5c04dfb3fda1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:28.539060 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb4dde9-dc34-4105-a8a1-5c04dfb3fda1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:28.541506 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb4dde9-dc34-4105-a8a1-5c04dfb3fda1 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:28.581444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1002/1965] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:59:28 2022] POST /identity/v3/projects => generated 359 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:59:28.585483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5208fae-8ced-494a-98bb-e661d580aa25 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:28.598133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5208fae-8ced-494a-98bb-e661d580aa25 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:28.680974 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab06c94f-98ee-4570-ad17-a36ee96d0d42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:28.682441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab06c94f-98ee-4570-ad17-a36ee96d0d42 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:28.682561 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab06c94f-98ee-4570-ad17-a36ee96d0d42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:28.682777 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab06c94f-98ee-4570-ad17-a36ee96d0d42 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:28.683445 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab06c94f-98ee-4570-ad17-a36ee96d0d42 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:28.685403 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab06c94f-98ee-4570-ad17-a36ee96d0d42 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:28.708853 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 964/1966] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:59:28 2022] POST /identity/v3/users => generated 280 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:59:28.713689 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb4dde9-dc34-4105-a8a1-5c04dfb3fda1 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:28.724446 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb4dde9-dc34-4105-a8a1-5c04dfb3fda1 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:28.730155 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48acae4e-4de4-43de-8f84-3bbdb0e01e15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:28.730957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48acae4e-4de4-43de-8f84-3bbdb0e01e15 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:28.731251 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48acae4e-4de4-43de-8f84-3bbdb0e01e15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:28.731620 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48acae4e-4de4-43de-8f84-3bbdb0e01e15 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:28.732185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48acae4e-4de4-43de-8f84-3bbdb0e01e15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:28.733803 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48acae4e-4de4-43de-8f84-3bbdb0e01e15 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:28.744462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1003/1967] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:59:28 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:59:28.748985 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab06c94f-98ee-4570-ad17-a36ee96d0d42 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:28.759481 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab06c94f-98ee-4570-ad17-a36ee96d0d42 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:28.764800 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8ebd98-e728-432b-9666-82d7e1b604ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:28.765407 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8ebd98-e728-432b-9666-82d7e1b604ba admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:28.765689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8ebd98-e728-432b-9666-82d7e1b604ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:28.765961 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8ebd98-e728-432b-9666-82d7e1b604ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:28.766605 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8ebd98-e728-432b-9666-82d7e1b604ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:28.768264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8ebd98-e728-432b-9666-82d7e1b604ba admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:28.779954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 965/1968] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:59:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:59:28.784331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48acae4e-4de4-43de-8f84-3bbdb0e01e15 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:28.794965 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48acae4e-4de4-43de-8f84-3bbdb0e01e15 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:28.799756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5eecc2-b0dc-4013-9972-5135f6adfdf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:28.800420 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5eecc2-b0dc-4013-9972-5135f6adfdf7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:28.800706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5eecc2-b0dc-4013-9972-5135f6adfdf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:28.800983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5eecc2-b0dc-4013-9972-5135f6adfdf7 admin admin] PATH_INFO: `/v3/projects/1a03a779af054a91b2672544374c6ef1/users/d2d01c06d1d44fab87c18e33e843f9ca/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:28.817080 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5eecc2-b0dc-4013-9972-5135f6adfdf7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a03a779af054a91b2672544374c6ef1, user_id=d2d01c06d1d44fab87c18e33e843f9ca, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:28.819793 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5eecc2-b0dc-4013-9972-5135f6adfdf7 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:28.834789 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1004/1969] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:59:28 2022] PUT /identity/v3/projects/1a03a779af054a91b2672544374c6ef1/users/d2d01c06d1d44fab87c18e33e843f9ca/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:28.846006 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773c09c2-dd74-488f-b3c7-a509d3391fe1 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:28.846285 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773c09c2-dd74-488f-b3c7-a509d3391fe1 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:28.846556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773c09c2-dd74-488f-b3c7-a509d3391fe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:28.877943 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-773c09c2-dd74-488f-b3c7-a509d3391fe1 None None] MFA Rules not processed for user `d2d01c06d1d44fab87c18e33e843f9ca`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:28.879219 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-773c09c2-dd74-488f-b3c7-a509d3391fe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:28.951231 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 966/1970] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:28 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:28.989366 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5eecc2-b0dc-4013-9972-5135f6adfdf7 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:28.991159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-5a5eecc2-b0dc-4013-9972-5135f6adfdf7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:29.014665 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:29.059239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5eecc2-b0dc-4013-9972-5135f6adfdf7 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:29.063240 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e820a97-e844-4af8-ab7d-b329e7b4ad25 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:29.063878 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e820a97-e844-4af8-ab7d-b329e7b4ad25 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:29.064163 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e820a97-e844-4af8-ab7d-b329e7b4ad25 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:29.064435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e820a97-e844-4af8-ab7d-b329e7b4ad25 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:29.074560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e820a97-e844-4af8-ab7d-b329e7b4ad25 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:29.076398 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e820a97-e844-4af8-ab7d-b329e7b4ad25 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:29.086657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1005/1971] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:28 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:59:29.103520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773c09c2-dd74-488f-b3c7-a509d3391fe1 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:29.124150 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:29.192222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773c09c2-dd74-488f-b3c7-a509d3391fe1 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:29.196867 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7e9d3a-bdff-4746-bd9e-a9c252492ce8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:29.198016 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7e9d3a-bdff-4746-bd9e-a9c252492ce8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:29.198296 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7e9d3a-bdff-4746-bd9e-a9c252492ce8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:29.198719 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7e9d3a-bdff-4746-bd9e-a9c252492ce8 admin admin] PATH_INFO: `/v3/users/d2d01c06d1d44fab87c18e33e843f9ca` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:29.199945 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7e9d3a-bdff-4746-bd9e-a9c252492ce8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d2d01c06d1d44fab87c18e33e843f9ca)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:29.202214 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7e9d3a-bdff-4746-bd9e-a9c252492ce8 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:29.251277 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5c7e9d3a-bdff-4746-bd9e-a9c252492ce8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd2d01c06d1d44fab87c18e33e843f9ca'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:29.251926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5c7e9d3a-bdff-4746-bd9e-a9c252492ce8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd2d01c06d1d44fab87c18e33e843f9ca'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:29.259344 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5c7e9d3a-bdff-4746-bd9e-a9c252492ce8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd2d01c06d1d44fab87c18e33e843f9ca'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:29.260611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5c7e9d3a-bdff-4746-bd9e-a9c252492ce8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd2d01c06d1d44fab87c18e33e843f9ca'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:29.272822 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5c7e9d3a-bdff-4746-bd9e-a9c252492ce8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd2d01c06d1d44fab87c18e33e843f9ca'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:29.285608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 967/1972] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:59:29 2022] DELETE /identity/v3/users/d2d01c06d1d44fab87c18e33e843f9ca => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:29.290492 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0906bbe-5931-4728-baa7-26ad5020e18c None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:29.290885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0906bbe-5931-4728-baa7-26ad5020e18c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:29.291193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0906bbe-5931-4728-baa7-26ad5020e18c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:29.309911 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-b0906bbe-5931-4728-baa7-26ad5020e18c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:29.311134 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b0906bbe-5931-4728-baa7-26ad5020e18c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:29.370290 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1006/1973] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:29.380130 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7e9d3a-bdff-4746-bd9e-a9c252492ce8 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:29.381543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5c7e9d3a-bdff-4746-bd9e-a9c252492ce8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:29.440811 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7e9d3a-bdff-4746-bd9e-a9c252492ce8 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:29.444519 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e9a732-ac58-40b3-95f4-64a855af8a19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:29.445138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e9a732-ac58-40b3-95f4-64a855af8a19 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:29.445420 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e9a732-ac58-40b3-95f4-64a855af8a19 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:29.445698 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e9a732-ac58-40b3-95f4-64a855af8a19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:29.456508 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e9a732-ac58-40b3-95f4-64a855af8a19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:29.457794 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e9a732-ac58-40b3-95f4-64a855af8a19 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:29.468683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 968/1974] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:59:29.746726 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0906bbe-5931-4728-baa7-26ad5020e18c None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:29.750481 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b0906bbe-5931-4728-baa7-26ad5020e18c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:29.783181 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0906bbe-5931-4728-baa7-26ad5020e18c None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:29.787894 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d2d359-5c6e-450d-a6b6-173981ac83f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:29.788535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d2d359-5c6e-450d-a6b6-173981ac83f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:29.788814 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d2d359-5c6e-450d-a6b6-173981ac83f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:29.789088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d2d359-5c6e-450d-a6b6-173981ac83f9 admin admin] PATH_INFO: `/v3/projects/1a03a779af054a91b2672544374c6ef1` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:29.790198 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d2d359-5c6e-450d-a6b6-173981ac83f9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1a03a779af054a91b2672544374c6ef1)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:29.792297 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d2d359-5c6e-450d-a6b6-173981ac83f9 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:29.857899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-88d2d359-5c6e-450d-a6b6-173981ac83f9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1a03a779af054a91b2672544374c6ef1', 'request_id': 'req-88d2d359-5c6e-450d-a6b6-173981ac83f9'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:29.869592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-88d2d359-5c6e-450d-a6b6-173981ac83f9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1a03a779af054a91b2672544374c6ef1', 'request_id': 'req-88d2d359-5c6e-450d-a6b6-173981ac83f9'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:29.877784 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-88d2d359-5c6e-450d-a6b6-173981ac83f9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1a03a779af054a91b2672544374c6ef1', 'request_id': 'req-88d2d359-5c6e-450d-a6b6-173981ac83f9'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:29.891566 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-88d2d359-5c6e-450d-a6b6-173981ac83f9 admin admin] The token cache is being invalidate because project 1a03a779af054a91b2672544374c6ef1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:59:29.891859 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-88d2d359-5c6e-450d-a6b6-173981ac83f9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:29.892540 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-88d2d359-5c6e-450d-a6b6-173981ac83f9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:29.895879 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1007/1975] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:59:29 2022] DELETE /identity/v3/projects/1a03a779af054a91b2672544374c6ef1 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:29.902244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e7ac8e-b4d7-4e76-9f79-b280bf36c06e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:29.902709 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e7ac8e-b4d7-4e76-9f79-b280bf36c06e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:29.902992 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e7ac8e-b4d7-4e76-9f79-b280bf36c06e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:29.928198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e5e7ac8e-b4d7-4e76-9f79-b280bf36c06e None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:29.929418 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e5e7ac8e-b4d7-4e76-9f79-b280bf36c06e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:29.967380 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:29.998564 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 969/1976] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:30.006494 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46489cf0-e7fa-450d-a834-1d366b1c9258 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:30.008276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46489cf0-e7fa-450d-a834-1d366b1c9258 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:30.008276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46489cf0-e7fa-450d-a834-1d366b1c9258 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:30.029086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-46489cf0-e7fa-450d-a834-1d366b1c9258 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:30.030535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-46489cf0-e7fa-450d-a834-1d366b1c9258 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:30.039332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1008/1977] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:30.044319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e7ac8e-b4d7-4e76-9f79-b280bf36c06e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:30.054513 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e7ac8e-b4d7-4e76-9f79-b280bf36c06e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:30.060101 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014080d2-2074-46d3-aae1-cb37530c1729 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:30.060746 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014080d2-2074-46d3-aae1-cb37530c1729 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:30.061025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014080d2-2074-46d3-aae1-cb37530c1729 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:30.061297 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014080d2-2074-46d3-aae1-cb37530c1729 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:30.061995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014080d2-2074-46d3-aae1-cb37530c1729 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:59:30.062233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014080d2-2074-46d3-aae1-cb37530c1729 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:30.063870 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014080d2-2074-46d3-aae1-cb37530c1729 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:30.079235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 970/1978] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:59:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:59:30.080496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46489cf0-e7fa-450d-a834-1d366b1c9258 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:30.090120 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46489cf0-e7fa-450d-a834-1d366b1c9258 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:30.094936 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8903bd8-63e2-4c04-a88d-61f256c91a1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:30.095549 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8903bd8-63e2-4c04-a88d-61f256c91a1a admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:30.096055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8903bd8-63e2-4c04-a88d-61f256c91a1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:30.096336 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8903bd8-63e2-4c04-a88d-61f256c91a1a admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:30.096974 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8903bd8-63e2-4c04-a88d-61f256c91a1a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:30.098982 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8903bd8-63e2-4c04-a88d-61f256c91a1a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:30.126880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1009/1979] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:59:30 2022] POST /identity/v3/projects => generated 373 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:59:30.131238 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014080d2-2074-46d3-aae1-cb37530c1729 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:30.141428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014080d2-2074-46d3-aae1-cb37530c1729 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:30.202639 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2195fe-7955-44c5-acc9-5d2b9b259dc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:30.203308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2195fe-7955-44c5-acc9-5d2b9b259dc0 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:30.205094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2195fe-7955-44c5-acc9-5d2b9b259dc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:30.205385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2195fe-7955-44c5-acc9-5d2b9b259dc0 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:30.206041 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2195fe-7955-44c5-acc9-5d2b9b259dc0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:30.208061 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2195fe-7955-44c5-acc9-5d2b9b259dc0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:30.234582 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 971/1980] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:59:30 2022] POST /identity/v3/users => generated 287 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:59:30.239202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8903bd8-63e2-4c04-a88d-61f256c91a1a admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:30.251936 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8903bd8-63e2-4c04-a88d-61f256c91a1a admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:30.256770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7e2b13-9012-4538-bf2c-6f5605777485 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:30.257385 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7e2b13-9012-4538-bf2c-6f5605777485 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:30.257675 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7e2b13-9012-4538-bf2c-6f5605777485 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:30.257982 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7e2b13-9012-4538-bf2c-6f5605777485 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:30.258626 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7e2b13-9012-4538-bf2c-6f5605777485 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:30.260287 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7e2b13-9012-4538-bf2c-6f5605777485 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:30.271256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1010/1981] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:59:30 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:59:30.275879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2195fe-7955-44c5-acc9-5d2b9b259dc0 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:30.285956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2195fe-7955-44c5-acc9-5d2b9b259dc0 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:30.291249 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85cafb8-dfea-45dd-8132-c8306644fab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:30.292418 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85cafb8-dfea-45dd-8132-c8306644fab4 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:30.292709 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85cafb8-dfea-45dd-8132-c8306644fab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:30.292988 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85cafb8-dfea-45dd-8132-c8306644fab4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:30.293644 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85cafb8-dfea-45dd-8132-c8306644fab4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:30.295302 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85cafb8-dfea-45dd-8132-c8306644fab4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:30.310128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 972/1982] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:59:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 00:59:30.311381 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7e2b13-9012-4538-bf2c-6f5605777485 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:30.321576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7e2b13-9012-4538-bf2c-6f5605777485 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:30.326612 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7c91e4-ac41-4169-9297-248aabcabc5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:30.327309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7c91e4-ac41-4169-9297-248aabcabc5a admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:30.327595 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7c91e4-ac41-4169-9297-248aabcabc5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:30.327872 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7c91e4-ac41-4169-9297-248aabcabc5a admin admin] PATH_INFO: `/v3/projects/92116166994646b39ec7a4c23e8cedc4/users/e04650eb7a9349e3b4f28ded86d26b43/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:30.330239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:30.347061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7c91e4-ac41-4169-9297-248aabcabc5a admin admin] RBAC: Authorizing `identity:create_grant(project_id=92116166994646b39ec7a4c23e8cedc4, user_id=e04650eb7a9349e3b4f28ded86d26b43, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:30.349744 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7c91e4-ac41-4169-9297-248aabcabc5a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:30.364476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1011/1983] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:59:30 2022] PUT /identity/v3/projects/92116166994646b39ec7a4c23e8cedc4/users/e04650eb7a9349e3b4f28ded86d26b43/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:30.369976 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8919790-4db8-4e4b-b85a-c561f49207c0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:30.370269 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8919790-4db8-4e4b-b85a-c561f49207c0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:30.370568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8919790-4db8-4e4b-b85a-c561f49207c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:30.389331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a8919790-4db8-4e4b-b85a-c561f49207c0 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:30.390524 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a8919790-4db8-4e4b-b85a-c561f49207c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:30.451046 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 973/1984] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:30.462054 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7c91e4-ac41-4169-9297-248aabcabc5a admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:30.464311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-ca7c91e4-ac41-4169-9297-248aabcabc5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:30.523378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7c91e4-ac41-4169-9297-248aabcabc5a admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:30.527621 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b5bafe-7b9f-4c98-bc7b-aba137b7a1fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:30.528240 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b5bafe-7b9f-4c98-bc7b-aba137b7a1fb service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:30.528526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b5bafe-7b9f-4c98-bc7b-aba137b7a1fb service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:30.528806 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b5bafe-7b9f-4c98-bc7b-aba137b7a1fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:30.538051 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b5bafe-7b9f-4c98-bc7b-aba137b7a1fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:30.539523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b5bafe-7b9f-4c98-bc7b-aba137b7a1fb service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:30.549882 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1012/1985] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:59:31.224105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e152d91-5ab3-4cf9-b1a3-374d19e31dc6 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:31.224641 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e152d91-5ab3-4cf9-b1a3-374d19e31dc6 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:31.224914 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e152d91-5ab3-4cf9-b1a3-374d19e31dc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:31.279984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-8e152d91-5ab3-4cf9-b1a3-374d19e31dc6 None None] MFA Rules not processed for user `e04650eb7a9349e3b4f28ded86d26b43`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:31.281263 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8e152d91-5ab3-4cf9-b1a3-374d19e31dc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:31.320583 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:31.356128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 974/1986] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:31 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:31.366227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b5bafe-7b9f-4c98-bc7b-aba137b7a1fb service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:31.367688 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d4b5bafe-7b9f-4c98-bc7b-aba137b7a1fb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:31.438329 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b5bafe-7b9f-4c98-bc7b-aba137b7a1fb service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:31.442435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04d6722-dcd5-430c-8beb-18944d4d3d2d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:31.443072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04d6722-dcd5-430c-8beb-18944d4d3d2d service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:31.443376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04d6722-dcd5-430c-8beb-18944d4d3d2d service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:31.443640 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04d6722-dcd5-430c-8beb-18944d4d3d2d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:31.453539 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04d6722-dcd5-430c-8beb-18944d4d3d2d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:31.455007 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04d6722-dcd5-430c-8beb-18944d4d3d2d service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:31.464954 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1013/1987] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:31 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:59:33.939213 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0285f71f-9a86-48b0-ade9-3dafa3671dca None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:33.939775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0285f71f-9a86-48b0-ade9-3dafa3671dca None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:33.939775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0285f71f-9a86-48b0-ade9-3dafa3671dca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:33.976988 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-0285f71f-9a86-48b0-ade9-3dafa3671dca None None] MFA Rules not processed for user `e04650eb7a9349e3b4f28ded86d26b43`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:33.979851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0285f71f-9a86-48b0-ade9-3dafa3671dca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:34.002201 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 975/1988] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:33 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:35.871393 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cc5149-efaf-4bed-9805-3bb6e67c8d89 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:35.871933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cc5149-efaf-4bed-9805-3bb6e67c8d89 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:35.871933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cc5149-efaf-4bed-9805-3bb6e67c8d89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:35.894693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-96cc5149-efaf-4bed-9805-3bb6e67c8d89 None None] MFA Rules not processed for user `466f2f8ddd184a98bdb5e922818905cc`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:35.896376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-96cc5149-efaf-4bed-9805-3bb6e67c8d89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:35.962294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1014/1989] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:35 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:38.556084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0285f71f-9a86-48b0-ade9-3dafa3671dca None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:38.558190 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0285f71f-9a86-48b0-ade9-3dafa3671dca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:38.572649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0285f71f-9a86-48b0-ade9-3dafa3671dca None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:38.578946 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed55995-ea35-48fa-839c-6efafa63626f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:38.579751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed55995-ea35-48fa-839c-6efafa63626f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:38.580099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed55995-ea35-48fa-839c-6efafa63626f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:38.580446 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed55995-ea35-48fa-839c-6efafa63626f admin admin] PATH_INFO: `/v3/users/aeada197e2224f8b936aa8ff2176a54e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:38.581668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed55995-ea35-48fa-839c-6efafa63626f admin admin] RBAC: Authorizing `identity:delete_user(user_id=aeada197e2224f8b936aa8ff2176a54e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:38.583778 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed55995-ea35-48fa-839c-6efafa63626f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:38.633034 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0ed55995-ea35-48fa-839c-6efafa63626f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aeada197e2224f8b936aa8ff2176a54e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:38.633693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0ed55995-ea35-48fa-839c-6efafa63626f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aeada197e2224f8b936aa8ff2176a54e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:38.641589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0ed55995-ea35-48fa-839c-6efafa63626f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aeada197e2224f8b936aa8ff2176a54e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:38.642341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0ed55995-ea35-48fa-839c-6efafa63626f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aeada197e2224f8b936aa8ff2176a54e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:38.655432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0ed55995-ea35-48fa-839c-6efafa63626f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aeada197e2224f8b936aa8ff2176a54e'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:38.667527 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 976/1990] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:59:38 2022] DELETE /identity/v3/users/aeada197e2224f8b936aa8ff2176a54e => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:38.671974 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365a7f56-4554-44c4-a7aa-981b0fbb32ff None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:38.672454 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365a7f56-4554-44c4-a7aa-981b0fbb32ff None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:38.672549 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365a7f56-4554-44c4-a7aa-981b0fbb32ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:38.691509 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-365a7f56-4554-44c4-a7aa-981b0fbb32ff None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:38.692647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-365a7f56-4554-44c4-a7aa-981b0fbb32ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:38.722836 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:38.754382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1015/1991] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:38.764694 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed55995-ea35-48fa-839c-6efafa63626f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:38.766097 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0ed55995-ea35-48fa-839c-6efafa63626f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:38.827730 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed55995-ea35-48fa-839c-6efafa63626f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:38.833328 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8a231f-d323-4016-99fe-83e9ad0ba07f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:38.833328 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8a231f-d323-4016-99fe-83e9ad0ba07f service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:38.833328 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8a231f-d323-4016-99fe-83e9ad0ba07f service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:38.833328 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8a231f-d323-4016-99fe-83e9ad0ba07f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:38.848544 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8a231f-d323-4016-99fe-83e9ad0ba07f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:38.850241 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8a231f-d323-4016-99fe-83e9ad0ba07f service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:38.860909 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 977/1992] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 00:59:39.443304 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365a7f56-4554-44c4-a7aa-981b0fbb32ff None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:39.445240 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-365a7f56-4554-44c4-a7aa-981b0fbb32ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:39.465165 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365a7f56-4554-44c4-a7aa-981b0fbb32ff None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:39.472497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0058569-b68c-403e-86f3-d50e5766ea48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:39.473485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0058569-b68c-403e-86f3-d50e5766ea48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:39.474241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0058569-b68c-403e-86f3-d50e5766ea48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:39.474768 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0058569-b68c-403e-86f3-d50e5766ea48 admin admin] PATH_INFO: `/v3/projects/c1d5006e67c24fdfb26bcfec851701d2` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:39.476859 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0058569-b68c-403e-86f3-d50e5766ea48 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c1d5006e67c24fdfb26bcfec851701d2)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:39.478849 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0058569-b68c-403e-86f3-d50e5766ea48 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:39.539751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b0058569-b68c-403e-86f3-d50e5766ea48 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c1d5006e67c24fdfb26bcfec851701d2', 'request_id': 'req-b0058569-b68c-403e-86f3-d50e5766ea48'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:39.548800 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b0058569-b68c-403e-86f3-d50e5766ea48 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c1d5006e67c24fdfb26bcfec851701d2', 'request_id': 'req-b0058569-b68c-403e-86f3-d50e5766ea48'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:39.555803 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b0058569-b68c-403e-86f3-d50e5766ea48 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c1d5006e67c24fdfb26bcfec851701d2', 'request_id': 'req-b0058569-b68c-403e-86f3-d50e5766ea48'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:39.570439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b0058569-b68c-403e-86f3-d50e5766ea48 admin admin] The token cache is being invalidate because project c1d5006e67c24fdfb26bcfec851701d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:59:39.570738 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b0058569-b68c-403e-86f3-d50e5766ea48 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:39.571470 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b0058569-b68c-403e-86f3-d50e5766ea48 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:39.573168 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1016/1993] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:59:39 2022] DELETE /identity/v3/projects/c1d5006e67c24fdfb26bcfec851701d2 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:39.578490 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f153b4-a525-4cf8-9fa7-e5a4e78382da None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:39.579232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f153b4-a525-4cf8-9fa7-e5a4e78382da None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:39.579232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f153b4-a525-4cf8-9fa7-e5a4e78382da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:39.609765 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d7f153b4-a525-4cf8-9fa7-e5a4e78382da None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:39.611076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d7f153b4-a525-4cf8-9fa7-e5a4e78382da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:39.683189 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:39.711382 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 978/1994] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:39.721340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf2b126-444a-4b4e-ac4e-a79ef580be0a None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:39.722243 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf2b126-444a-4b4e-ac4e-a79ef580be0a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:39.722846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf2b126-444a-4b4e-ac4e-a79ef580be0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:39.752152 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-acf2b126-444a-4b4e-ac4e-a79ef580be0a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:39.753593 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-acf2b126-444a-4b4e-ac4e-a79ef580be0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:39.769012 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1017/1995] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:39.770693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f153b4-a525-4cf8-9fa7-e5a4e78382da None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:39.782560 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f153b4-a525-4cf8-9fa7-e5a4e78382da None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:39.788945 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfe9e42-644a-4884-8f7b-d15872244bc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:39.789635 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfe9e42-644a-4884-8f7b-d15872244bc1 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:39.789924 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfe9e42-644a-4884-8f7b-d15872244bc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:39.790196 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfe9e42-644a-4884-8f7b-d15872244bc1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:39.790892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfe9e42-644a-4884-8f7b-d15872244bc1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:59:39.791196 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfe9e42-644a-4884-8f7b-d15872244bc1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:39.793807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfe9e42-644a-4884-8f7b-d15872244bc1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:39.816008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 979/1996] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:59:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:59:39.817376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf2b126-444a-4b4e-ac4e-a79ef580be0a None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:39.828397 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf2b126-444a-4b4e-ac4e-a79ef580be0a None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:39.833327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8001563-7cd1-4390-ad2f-0a1cd5154bef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:39.833878 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8001563-7cd1-4390-ad2f-0a1cd5154bef admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:39.834159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8001563-7cd1-4390-ad2f-0a1cd5154bef admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:39.834431 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8001563-7cd1-4390-ad2f-0a1cd5154bef admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:39.835228 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8001563-7cd1-4390-ad2f-0a1cd5154bef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:39.837143 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8001563-7cd1-4390-ad2f-0a1cd5154bef admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:39.869145 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1018/1997] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:59:39 2022] POST /identity/v3/projects => generated 369 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:59:39.873760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfe9e42-644a-4884-8f7b-d15872244bc1 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:39.889064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfe9e42-644a-4884-8f7b-d15872244bc1 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:39.962022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38ee34c-b51d-493e-97f1-9df6fe14860d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:39.962714 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38ee34c-b51d-493e-97f1-9df6fe14860d admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:39.963110 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38ee34c-b51d-493e-97f1-9df6fe14860d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:39.963341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38ee34c-b51d-493e-97f1-9df6fe14860d admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:39.964164 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38ee34c-b51d-493e-97f1-9df6fe14860d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:39.966119 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38ee34c-b51d-493e-97f1-9df6fe14860d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:39.991084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 980/1998] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:59:39 2022] POST /identity/v3/users => generated 285 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:59:39.998942 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8001563-7cd1-4390-ad2f-0a1cd5154bef admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:40.000080 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38ee34c-b51d-493e-97f1-9df6fe14860d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:40.002709 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d38ee34c-b51d-493e-97f1-9df6fe14860d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:40.007128 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8001563-7cd1-4390-ad2f-0a1cd5154bef admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:40.014645 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d96ed8-8d5a-422e-af13-1953741583cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:40.014887 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d96ed8-8d5a-422e-af13-1953741583cf admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:40.015122 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d96ed8-8d5a-422e-af13-1953741583cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:40.015795 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d96ed8-8d5a-422e-af13-1953741583cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:40.016456 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d96ed8-8d5a-422e-af13-1953741583cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:40.016837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38ee34c-b51d-493e-97f1-9df6fe14860d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:40.018244 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d96ed8-8d5a-422e-af13-1953741583cf admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:40.023670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e2eac5-9ac4-4060-af0c-1ba08281352a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:40.024413 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e2eac5-9ac4-4060-af0c-1ba08281352a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:40.024604 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e2eac5-9ac4-4060-af0c-1ba08281352a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:40.024886 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e2eac5-9ac4-4060-af0c-1ba08281352a admin admin] PATH_INFO: `/v3/users/ff4a9c6cf0a54f4a98ef180ae6402ab3` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:40.026117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e2eac5-9ac4-4060-af0c-1ba08281352a admin admin] RBAC: Authorizing `identity:delete_user(user_id=ff4a9c6cf0a54f4a98ef180ae6402ab3)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:40.028955 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e2eac5-9ac4-4060-af0c-1ba08281352a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:40.032383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1019/1999] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:59:39 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:59:40.036026 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d96ed8-8d5a-422e-af13-1953741583cf admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:40.049310 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d96ed8-8d5a-422e-af13-1953741583cf admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:40.055197 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2549af1-0c1e-47c7-9c0a-36f6303b6b73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:40.056730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2549af1-0c1e-47c7-9c0a-36f6303b6b73 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:40.057911 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2549af1-0c1e-47c7-9c0a-36f6303b6b73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:40.058556 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2549af1-0c1e-47c7-9c0a-36f6303b6b73 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:40.059428 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2549af1-0c1e-47c7-9c0a-36f6303b6b73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:40.061243 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2549af1-0c1e-47c7-9c0a-36f6303b6b73 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:40.077471 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1020/2000] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:59:40 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 00:59:40.081661 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2549af1-0c1e-47c7-9c0a-36f6303b6b73 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:40.097118 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2549af1-0c1e-47c7-9c0a-36f6303b6b73 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:40.100444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-00e2eac5-9ac4-4060-af0c-1ba08281352a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ff4a9c6cf0a54f4a98ef180ae6402ab3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:40.101738 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-00e2eac5-9ac4-4060-af0c-1ba08281352a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ff4a9c6cf0a54f4a98ef180ae6402ab3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:40.106618 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982f474f-d3a8-4009-b772-21223e12b4b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:40.107662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982f474f-d3a8-4009-b772-21223e12b4b5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:40.108207 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982f474f-d3a8-4009-b772-21223e12b4b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:40.108847 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982f474f-d3a8-4009-b772-21223e12b4b5 admin admin] PATH_INFO: `/v3/projects/486aaad8b7324b38b38eba267348eed8/users/9e18517e7b944110a0116e5bb4ae3bc8/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:40.115927 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-00e2eac5-9ac4-4060-af0c-1ba08281352a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ff4a9c6cf0a54f4a98ef180ae6402ab3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:40.117210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-00e2eac5-9ac4-4060-af0c-1ba08281352a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ff4a9c6cf0a54f4a98ef180ae6402ab3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:40.134928 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-00e2eac5-9ac4-4060-af0c-1ba08281352a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ff4a9c6cf0a54f4a98ef180ae6402ab3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:40.138697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982f474f-d3a8-4009-b772-21223e12b4b5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=486aaad8b7324b38b38eba267348eed8, user_id=9e18517e7b944110a0116e5bb4ae3bc8, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:40.144822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982f474f-d3a8-4009-b772-21223e12b4b5 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:40.151728 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 981/2001] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 00:59:39 2022] DELETE /identity/v3/users/ff4a9c6cf0a54f4a98ef180ae6402ab3 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 00:59:40.156575 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd66d85-5555-4187-89b3-8994484f4f6c None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:40.156864 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd66d85-5555-4187-89b3-8994484f4f6c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:40.157140 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd66d85-5555-4187-89b3-8994484f4f6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:40.182894 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1021/2002] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:59:40 2022] PUT /identity/v3/projects/486aaad8b7324b38b38eba267348eed8/users/9e18517e7b944110a0116e5bb4ae3bc8/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:40.188320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9e3090-e8f2-406a-a09c-85877721e904 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:40.188615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9e3090-e8f2-406a-a09c-85877721e904 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:40.189061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9e3090-e8f2-406a-a09c-85877721e904 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:40.202437 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-cdd66d85-5555-4187-89b3-8994484f4f6c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:40.205098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cdd66d85-5555-4187-89b3-8994484f4f6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:40.248779 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-cb9e3090-e8f2-406a-a09c-85877721e904 None None] MFA Rules not processed for user `9e18517e7b944110a0116e5bb4ae3bc8`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:40.250925 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-cb9e3090-e8f2-406a-a09c-85877721e904 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:40.257710 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:40.290276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:40.294258 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 982/2003] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:40.306459 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd66d85-5555-4187-89b3-8994484f4f6c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:40.308463 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cdd66d85-5555-4187-89b3-8994484f4f6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:40.365262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1022/2004] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:40 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:40.381638 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9e3090-e8f2-406a-a09c-85877721e904 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:40.383109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-cb9e3090-e8f2-406a-a09c-85877721e904 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:40.416642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd66d85-5555-4187-89b3-8994484f4f6c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:40.421219 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e91c0a7-3b90-44da-9f53-e1e3aa37603f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:40.421885 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e91c0a7-3b90-44da-9f53-e1e3aa37603f service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:40.422198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e91c0a7-3b90-44da-9f53-e1e3aa37603f service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:40.422720 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e91c0a7-3b90-44da-9f53-e1e3aa37603f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:40.441229 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e91c0a7-3b90-44da-9f53-e1e3aa37603f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:40.443054 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e91c0a7-3b90-44da-9f53-e1e3aa37603f service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:40.454626 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 983/2005] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:59:40.462150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9e3090-e8f2-406a-a09c-85877721e904 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:40.466839 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81176e81-6e8f-40c7-880c-a000dc11e06d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '78473c29cef449b89bc9185de67f1bf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:40.467500 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81176e81-6e8f-40c7-880c-a000dc11e06d service glance] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:40.467786 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81176e81-6e8f-40c7-880c-a000dc11e06d service glance] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:40.468063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81176e81-6e8f-40c7-880c-a000dc11e06d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:40.477611 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81176e81-6e8f-40c7-880c-a000dc11e06d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:40.479158 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81176e81-6e8f-40c7-880c-a000dc11e06d service glance] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:40.489157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1023/2006] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:40 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:59:40.858749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e91c0a7-3b90-44da-9f53-e1e3aa37603f service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:40.860391 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1e91c0a7-3b90-44da-9f53-e1e3aa37603f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:40.886659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e91c0a7-3b90-44da-9f53-e1e3aa37603f service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:40.900087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e95476-c797-4da2-a7a6-8eea02251b6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:40.903642 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e95476-c797-4da2-a7a6-8eea02251b6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:40.906887 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e95476-c797-4da2-a7a6-8eea02251b6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:40.915505 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e95476-c797-4da2-a7a6-8eea02251b6f admin admin] PATH_INFO: `/v3/projects/411d2dda80534b15946c38b816defaa1` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:40.920928 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e95476-c797-4da2-a7a6-8eea02251b6f admin admin] RBAC: Authorizing `identity:delete_project(project_id=411d2dda80534b15946c38b816defaa1)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:40.927390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e95476-c797-4da2-a7a6-8eea02251b6f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:41.083750 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d8e95476-c797-4da2-a7a6-8eea02251b6f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '411d2dda80534b15946c38b816defaa1', 'request_id': 'req-d8e95476-c797-4da2-a7a6-8eea02251b6f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:41.099953 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d8e95476-c797-4da2-a7a6-8eea02251b6f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '411d2dda80534b15946c38b816defaa1', 'request_id': 'req-d8e95476-c797-4da2-a7a6-8eea02251b6f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:41.120350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d8e95476-c797-4da2-a7a6-8eea02251b6f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '411d2dda80534b15946c38b816defaa1', 'request_id': 'req-d8e95476-c797-4da2-a7a6-8eea02251b6f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:41.147491 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d8e95476-c797-4da2-a7a6-8eea02251b6f admin admin] The token cache is being invalidate because project 411d2dda80534b15946c38b816defaa1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 00:59:41.147803 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d8e95476-c797-4da2-a7a6-8eea02251b6f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:41.148460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-d8e95476-c797-4da2-a7a6-8eea02251b6f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 00:59:41.150172 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 984/2007] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 00:59:40 2022] DELETE /identity/v3/projects/411d2dda80534b15946c38b816defaa1 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:41.156906 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675e03a3-2f33-49c6-b405-faf218c5b244 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:41.157319 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675e03a3-2f33-49c6-b405-faf218c5b244 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:41.157455 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675e03a3-2f33-49c6-b405-faf218c5b244 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:41.205495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-675e03a3-2f33-49c6-b405-faf218c5b244 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:41.208815 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-675e03a3-2f33-49c6-b405-faf218c5b244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:41.366190 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1024/2008] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:41.375153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afeee38-48e8-4227-b2ca-9d02ee7e7b5b None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:41.375441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afeee38-48e8-4227-b2ca-9d02ee7e7b5b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:41.375712 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afeee38-48e8-4227-b2ca-9d02ee7e7b5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:41.405874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3afeee38-48e8-4227-b2ca-9d02ee7e7b5b None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:41.407825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3afeee38-48e8-4227-b2ca-9d02ee7e7b5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:41.428965 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 985/2009] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:41.429180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675e03a3-2f33-49c6-b405-faf218c5b244 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:41.439830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675e03a3-2f33-49c6-b405-faf218c5b244 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:41.453530 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63be3f93-0c5c-4d8b-af4a-57e52c26eb39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:41.454421 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63be3f93-0c5c-4d8b-af4a-57e52c26eb39 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:41.454936 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63be3f93-0c5c-4d8b-af4a-57e52c26eb39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:41.455519 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63be3f93-0c5c-4d8b-af4a-57e52c26eb39 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:41.456568 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63be3f93-0c5c-4d8b-af4a-57e52c26eb39 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 00:59:41.457335 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63be3f93-0c5c-4d8b-af4a-57e52c26eb39 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:41.459204 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63be3f93-0c5c-4d8b-af4a-57e52c26eb39 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:41.503109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1025/2010] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 00:59:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 00:59:41.506611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afeee38-48e8-4227-b2ca-9d02ee7e7b5b None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:41.517569 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afeee38-48e8-4227-b2ca-9d02ee7e7b5b None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:41.524604 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901384a6-c568-443c-9f3a-3535857ad31c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:41.525368 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901384a6-c568-443c-9f3a-3535857ad31c admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:41.526024 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901384a6-c568-443c-9f3a-3535857ad31c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:41.526024 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901384a6-c568-443c-9f3a-3535857ad31c admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:41.526759 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901384a6-c568-443c-9f3a-3535857ad31c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:41.528903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901384a6-c568-443c-9f3a-3535857ad31c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:41.571342 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 986/2011] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:59:41 2022] POST /identity/v3/projects => generated 379 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:59:41.576279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63be3f93-0c5c-4d8b-af4a-57e52c26eb39 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:41.591644 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63be3f93-0c5c-4d8b-af4a-57e52c26eb39 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:41.729065 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8684130f-448b-41ef-9c54-221d5669a085 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:41.729732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8684130f-448b-41ef-9c54-221d5669a085 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:41.730014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8684130f-448b-41ef-9c54-221d5669a085 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:41.730279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8684130f-448b-41ef-9c54-221d5669a085 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:41.731554 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8684130f-448b-41ef-9c54-221d5669a085 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:41.735160 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8684130f-448b-41ef-9c54-221d5669a085 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:41.748075 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:41.777631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901384a6-c568-443c-9f3a-3535857ad31c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:41.778968 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1026/2012] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:59:41 2022] POST /identity/v3/users => generated 290 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:59:41.812284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901384a6-c568-443c-9f3a-3535857ad31c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:41.826496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6003ed-2eb3-4716-8855-b080753b3697 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:41.828107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6003ed-2eb3-4716-8855-b080753b3697 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:41.828742 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6003ed-2eb3-4716-8855-b080753b3697 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:41.829360 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6003ed-2eb3-4716-8855-b080753b3697 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:41.830118 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6003ed-2eb3-4716-8855-b080753b3697 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:41.832070 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6003ed-2eb3-4716-8855-b080753b3697 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:41.844412 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 987/2013] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:59:41 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:59:41.849095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8684130f-448b-41ef-9c54-221d5669a085 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:41.879347 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8684130f-448b-41ef-9c54-221d5669a085 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:41.886458 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea591f2-6bb9-426e-9455-958193d9ca91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:41.887462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea591f2-6bb9-426e-9455-958193d9ca91 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:41.888095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea591f2-6bb9-426e-9455-958193d9ca91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:41.888556 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea591f2-6bb9-426e-9455-958193d9ca91 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:41.889553 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea591f2-6bb9-426e-9455-958193d9ca91 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:41.892066 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea591f2-6bb9-426e-9455-958193d9ca91 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:41.914173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1027/2014] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:59:41 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:59:41.916743 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6003ed-2eb3-4716-8855-b080753b3697 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:41.933070 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6003ed-2eb3-4716-8855-b080753b3697 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:41.940461 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e50307-e897-4855-98b8-3976cb73999c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:41.941494 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e50307-e897-4855-98b8-3976cb73999c admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:41.942008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e50307-e897-4855-98b8-3976cb73999c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:41.944187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e50307-e897-4855-98b8-3976cb73999c admin admin] PATH_INFO: `/v3/projects/cb1743385ea14cc39244c83e8fc88e7e/users/c4e65cef89af48a7a51e5548634de75f/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:42.003298 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e50307-e897-4855-98b8-3976cb73999c admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb1743385ea14cc39244c83e8fc88e7e, user_id=c4e65cef89af48a7a51e5548634de75f, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:42.006995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e50307-e897-4855-98b8-3976cb73999c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:42.013265 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:42.027386 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 988/2015] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:59:41 2022] PUT /identity/v3/projects/cb1743385ea14cc39244c83e8fc88e7e/users/c4e65cef89af48a7a51e5548634de75f/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:42.033540 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0f498a-d2dd-41d5-9c76-974d49cda365 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:42.033820 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0f498a-d2dd-41d5-9c76-974d49cda365 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:42.034094 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0f498a-d2dd-41d5-9c76-974d49cda365 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:42.054091 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-4c0f498a-d2dd-41d5-9c76-974d49cda365 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:42.055338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4c0f498a-d2dd-41d5-9c76-974d49cda365 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:42.129769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1028/2016] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:42.141174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e50307-e897-4855-98b8-3976cb73999c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:42.143325 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-07e50307-e897-4855-98b8-3976cb73999c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:42.217471 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e50307-e897-4855-98b8-3976cb73999c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:42.222242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc0b9d2-01ce-4693-800b-f2452e4830ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:42.223141 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc0b9d2-01ce-4693-800b-f2452e4830ae service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:42.223387 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc0b9d2-01ce-4693-800b-f2452e4830ae service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:42.223814 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc0b9d2-01ce-4693-800b-f2452e4830ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:42.237736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc0b9d2-01ce-4693-800b-f2452e4830ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:42.239909 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc0b9d2-01ce-4693-800b-f2452e4830ae service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:42.264416 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 989/2017] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 00:59:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 00:59:46.793807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a73b26-1e24-49b3-a0df-3a66a1d77f39 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:46.794358 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a73b26-1e24-49b3-a0df-3a66a1d77f39 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:46.794584 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a73b26-1e24-49b3-a0df-3a66a1d77f39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:46.825401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-27a73b26-1e24-49b3-a0df-3a66a1d77f39 None None] MFA Rules not processed for user `33b136afbf8a4d98a9d62053b68707e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:46.826825 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-27a73b26-1e24-49b3-a0df-3a66a1d77f39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:46.907117 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1029/2018] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 16 00:59:46 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:46.936506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc0b9d2-01ce-4693-800b-f2452e4830ae service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:46.938578 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4dc0b9d2-01ce-4693-800b-f2452e4830ae service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:47.039877 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d727e771-2ca8-4ab7-9888-919a2c1d6907 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:47.040176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d727e771-2ca8-4ab7-9888-919a2c1d6907 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:47.040437 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d727e771-2ca8-4ab7-9888-919a2c1d6907 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:47.072441 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-d727e771-2ca8-4ab7-9888-919a2c1d6907 None None] MFA Rules not processed for user `b01ce2d27bdb4c91a3f55c891f949e00`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:47.073767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d727e771-2ca8-4ab7-9888-919a2c1d6907 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:47.080158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc0b9d2-01ce-4693-800b-f2452e4830ae service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:47.086330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d16a7a3-9d2e-4740-9179-da9ff8b73356 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de64d498da78448394b0d4d1851a2789', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:47.087206 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d16a7a3-9d2e-4740-9179-da9ff8b73356 service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:47.087551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d16a7a3-9d2e-4740-9179-da9ff8b73356 service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:47.092997 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d16a7a3-9d2e-4740-9179-da9ff8b73356 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:47.103314 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d16a7a3-9d2e-4740-9179-da9ff8b73356 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:47.106890 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d16a7a3-9d2e-4740-9179-da9ff8b73356 service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:47.119078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 990/2019] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 16 00:59:46 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 16 00:59:47.208436 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1030/2020] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:47 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:50.300747 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75006db-3029-411f-9bc0-38ceb0c9b2f8 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:50.307990 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75006db-3029-411f-9bc0-38ceb0c9b2f8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:50.308405 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75006db-3029-411f-9bc0-38ceb0c9b2f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:50.356494 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c75006db-3029-411f-9bc0-38ceb0c9b2f8 None None] MFA Rules not processed for user `c4e65cef89af48a7a51e5548634de75f`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 00:59:50.357593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c75006db-3029-411f-9bc0-38ceb0c9b2f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 00:59:50.480025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 991/2021] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 00:59:50 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 00:59:50.481204 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d727e771-2ca8-4ab7-9888-919a2c1d6907 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:50.495947 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d727e771-2ca8-4ab7-9888-919a2c1d6907 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:50.504129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed49772-2830-4a6e-91e4-7246f6e060e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:50.504762 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed49772-2830-4a6e-91e4-7246f6e060e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:50.505244 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed49772-2830-4a6e-91e4-7246f6e060e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:50.505528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed49772-2830-4a6e-91e4-7246f6e060e6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:50.506188 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed49772-2830-4a6e-91e4-7246f6e060e6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:50.508204 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed49772-2830-4a6e-91e4-7246f6e060e6 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:50.558491 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75006db-3029-411f-9bc0-38ceb0c9b2f8 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:50.559146 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1031/2022] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 00:59:50 2022] POST /identity/v3/projects => generated 379 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 00:59:50.575673 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75006db-3029-411f-9bc0-38ceb0c9b2f8 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:50.598660 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:50.681308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901b2a40-959e-4db4-b4f8-4cc9acb76a1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:50.682087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901b2a40-959e-4db4-b4f8-4cc9acb76a1c admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:50.682791 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901b2a40-959e-4db4-b4f8-4cc9acb76a1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:50.683100 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901b2a40-959e-4db4-b4f8-4cc9acb76a1c admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:50.683760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901b2a40-959e-4db4-b4f8-4cc9acb76a1c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:50.685726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901b2a40-959e-4db4-b4f8-4cc9acb76a1c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:50.714416 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 992/2023] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 00:59:50 2022] POST /identity/v3/users => generated 296 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 00:59:50.718905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed49772-2830-4a6e-91e4-7246f6e060e6 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:50.730771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed49772-2830-4a6e-91e4-7246f6e060e6 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:50.737538 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6482c5d2-f4b2-4d1e-9554-c78d38dc6802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:50.738176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6482c5d2-f4b2-4d1e-9554-c78d38dc6802 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:50.738461 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6482c5d2-f4b2-4d1e-9554-c78d38dc6802 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:50.738752 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6482c5d2-f4b2-4d1e-9554-c78d38dc6802 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:50.739587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6482c5d2-f4b2-4d1e-9554-c78d38dc6802 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:50.741237 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6482c5d2-f4b2-4d1e-9554-c78d38dc6802 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:50.743997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 00:59:50.758920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1032/2024] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:59:50 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:59:50.762334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901b2a40-959e-4db4-b4f8-4cc9acb76a1c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:50.775019 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901b2a40-959e-4db4-b4f8-4cc9acb76a1c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:50.781050 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fe0fe6-c1dc-460d-970b-f14808d1feea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:50.781726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fe0fe6-c1dc-460d-970b-f14808d1feea admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:50.782022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fe0fe6-c1dc-460d-970b-f14808d1feea admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:50.782282 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fe0fe6-c1dc-460d-970b-f14808d1feea admin admin] PATH_INFO: `/v3/projects/403b173e9ff04603b1c33837dcd1ba73/users/e4c1f5685bf840588d90104aa11e4920/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:50.807511 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fe0fe6-c1dc-460d-970b-f14808d1feea admin admin] RBAC: Authorizing `identity:create_grant(project_id=403b173e9ff04603b1c33837dcd1ba73, user_id=e4c1f5685bf840588d90104aa11e4920, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:50.810908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fe0fe6-c1dc-460d-970b-f14808d1feea admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:50.832808 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 993/2025] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:59:50 2022] PUT /identity/v3/projects/403b173e9ff04603b1c33837dcd1ba73/users/e4c1f5685bf840588d90104aa11e4920/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 00:59:50.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6482c5d2-f4b2-4d1e-9554-c78d38dc6802 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:50.852503 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6482c5d2-f4b2-4d1e-9554-c78d38dc6802 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:51.017075 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e089c67-6686-4f72-93aa-f68c4a6f3ee3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:51.017809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e089c67-6686-4f72-93aa-f68c4a6f3ee3 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:51.018109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e089c67-6686-4f72-93aa-f68c4a6f3ee3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:51.018362 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e089c67-6686-4f72-93aa-f68c4a6f3ee3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:51.019566 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e089c67-6686-4f72-93aa-f68c4a6f3ee3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:51.021545 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e089c67-6686-4f72-93aa-f68c4a6f3ee3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:51.040714 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1033/2026] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 00:59:50 2022] GET /identity/v3/roles => generated 1420 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 00:59:51.045841 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fe0fe6-c1dc-460d-970b-f14808d1feea admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 00:59:51.061898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fe0fe6-c1dc-460d-970b-f14808d1feea admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 00:59:51.068212 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b494e977-3211-4c8f-9ee8-627bf0a74c83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 00:59:51.069302 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b494e977-3211-4c8f-9ee8-627bf0a74c83 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 00:59:51.069782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b494e977-3211-4c8f-9ee8-627bf0a74c83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 00:59:51.070258 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b494e977-3211-4c8f-9ee8-627bf0a74c83 admin admin] PATH_INFO: `/v3/projects/403b173e9ff04603b1c33837dcd1ba73/users/e4c1f5685bf840588d90104aa11e4920/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 00:59:51.073335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b494e977-3211-4c8f-9ee8-627bf0a74c83 admin admin] RBAC: Authorizing `identity:create_grant(project_id=403b173e9ff04603b1c33837dcd1ba73, user_id=e4c1f5685bf840588d90104aa11e4920, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 00:59:51.077067 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b494e977-3211-4c8f-9ee8-627bf0a74c83 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 00:59:51.091322 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 994/2027] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 00:59:51 2022] PUT /identity/v3/projects/403b173e9ff04603b1c33837dcd1ba73/users/e4c1f5685bf840588d90104aa11e4920/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:00:01.032428 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=30 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:00:01.085837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=36,keystone:INSERT=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:00:01.091337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533a0008-5cf0-4bec-bb21-08aee9395adf None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:01.091574 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533a0008-5cf0-4bec-bb21-08aee9395adf None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:01.091870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533a0008-5cf0-4bec-bb21-08aee9395adf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:01.143989 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-533a0008-5cf0-4bec-bb21-08aee9395adf None None] MFA Rules not processed for user `e04650eb7a9349e3b4f28ded86d26b43`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:00:01.145353 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-533a0008-5cf0-4bec-bb21-08aee9395adf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:01.225131 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1034/2028] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:00:01 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:00:03.502979 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ba5f74-50e3-47f0-8e64-8266e3e0ff67 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:03.503595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ba5f74-50e3-47f0-8e64-8266e3e0ff67 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:03.503595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ba5f74-50e3-47f0-8e64-8266e3e0ff67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:03.581655 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-59ba5f74-50e3-47f0-8e64-8266e3e0ff67 None None] MFA Rules not processed for user `e4c1f5685bf840588d90104aa11e4920`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:00:03.583357 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-59ba5f74-50e3-47f0-8e64-8266e3e0ff67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:03.822903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 995/2029] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:00:03 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 321 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:00:03.896756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533a0008-5cf0-4bec-bb21-08aee9395adf None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:03.899218 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-533a0008-5cf0-4bec-bb21-08aee9395adf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:04.096884 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533a0008-5cf0-4bec-bb21-08aee9395adf None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:04.115480 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1a7da2-646e-4b3f-85f6-8c540f48d280 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:04.116136 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1a7da2-646e-4b3f-85f6-8c540f48d280 service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:04.116419 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1a7da2-646e-4b3f-85f6-8c540f48d280 service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:04.116693 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1a7da2-646e-4b3f-85f6-8c540f48d280 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:04.237835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1a7da2-646e-4b3f-85f6-8c540f48d280 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:04.239501 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1a7da2-646e-4b3f-85f6-8c540f48d280 service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:04.265764 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1035/2030] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:00:03 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:00:11.412047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ba5f74-50e3-47f0-8e64-8266e3e0ff67 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:11.413506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-59ba5f74-50e3-47f0-8e64-8266e3e0ff67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:11.498176 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ba5f74-50e3-47f0-8e64-8266e3e0ff67 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:11.502958 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0503fdda-ad72-454d-8a01-1a06e6b1c944 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:11.503615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0503fdda-ad72-454d-8a01-1a06e6b1c944 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:11.503972 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0503fdda-ad72-454d-8a01-1a06e6b1c944 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:11.504251 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0503fdda-ad72-454d-8a01-1a06e6b1c944 admin admin] PATH_INFO: `/v3/users/9e18517e7b944110a0116e5bb4ae3bc8` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:11.505341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0503fdda-ad72-454d-8a01-1a06e6b1c944 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9e18517e7b944110a0116e5bb4ae3bc8)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:11.507335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0503fdda-ad72-454d-8a01-1a06e6b1c944 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:11.562047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0503fdda-ad72-454d-8a01-1a06e6b1c944 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9e18517e7b944110a0116e5bb4ae3bc8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:00:11.562773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0503fdda-ad72-454d-8a01-1a06e6b1c944 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9e18517e7b944110a0116e5bb4ae3bc8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:00:11.572627 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0503fdda-ad72-454d-8a01-1a06e6b1c944 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9e18517e7b944110a0116e5bb4ae3bc8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:00:11.573328 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0503fdda-ad72-454d-8a01-1a06e6b1c944 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9e18517e7b944110a0116e5bb4ae3bc8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:00:11.589225 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-0503fdda-ad72-454d-8a01-1a06e6b1c944 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9e18517e7b944110a0116e5bb4ae3bc8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:00:11.605358 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 996/2031] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:00:11 2022] DELETE /identity/v3/users/9e18517e7b944110a0116e5bb4ae3bc8 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:00:11.610068 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314115d8-9885-4778-bc21-cd9d7f5ce21d None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:11.610627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314115d8-9885-4778-bc21-cd9d7f5ce21d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:11.610627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314115d8-9885-4778-bc21-cd9d7f5ce21d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:11.631541 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-314115d8-9885-4778-bc21-cd9d7f5ce21d None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:00:11.632759 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-314115d8-9885-4778-bc21-cd9d7f5ce21d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:11.679898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:00:11.708788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1036/2032] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:00:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 01:00:11.718766 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0503fdda-ad72-454d-8a01-1a06e6b1c944 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:11.720441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0503fdda-ad72-454d-8a01-1a06e6b1c944 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:11.730796 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:00:11.793725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0503fdda-ad72-454d-8a01-1a06e6b1c944 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:11.797659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305a8fdf-4721-475e-b9c3-75f3397ee32d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:11.798293 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305a8fdf-4721-475e-b9c3-75f3397ee32d service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:11.799247 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305a8fdf-4721-475e-b9c3-75f3397ee32d service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:11.799538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305a8fdf-4721-475e-b9c3-75f3397ee32d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:11.810471 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305a8fdf-4721-475e-b9c3-75f3397ee32d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:11.811959 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305a8fdf-4721-475e-b9c3-75f3397ee32d service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:11.822619 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 997/2033] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:00:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:00:12.124699 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314115d8-9885-4778-bc21-cd9d7f5ce21d None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:12.126219 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-314115d8-9885-4778-bc21-cd9d7f5ce21d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:12.141251 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314115d8-9885-4778-bc21-cd9d7f5ce21d None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:12.146025 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896ac805-a745-4d25-bc24-2dcfa0384358 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:12.146836 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896ac805-a745-4d25-bc24-2dcfa0384358 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:12.147272 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896ac805-a745-4d25-bc24-2dcfa0384358 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:12.147666 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896ac805-a745-4d25-bc24-2dcfa0384358 admin admin] PATH_INFO: `/v3/projects/486aaad8b7324b38b38eba267348eed8` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:12.148901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896ac805-a745-4d25-bc24-2dcfa0384358 admin admin] RBAC: Authorizing `identity:delete_project(project_id=486aaad8b7324b38b38eba267348eed8)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:12.151583 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896ac805-a745-4d25-bc24-2dcfa0384358 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:12.202756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-896ac805-a745-4d25-bc24-2dcfa0384358 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '486aaad8b7324b38b38eba267348eed8', 'request_id': 'req-896ac805-a745-4d25-bc24-2dcfa0384358'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:00:12.213675 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-896ac805-a745-4d25-bc24-2dcfa0384358 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '486aaad8b7324b38b38eba267348eed8', 'request_id': 'req-896ac805-a745-4d25-bc24-2dcfa0384358'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:00:12.220891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-896ac805-a745-4d25-bc24-2dcfa0384358 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '486aaad8b7324b38b38eba267348eed8', 'request_id': 'req-896ac805-a745-4d25-bc24-2dcfa0384358'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:00:12.233410 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-896ac805-a745-4d25-bc24-2dcfa0384358 admin admin] The token cache is being invalidate because project 486aaad8b7324b38b38eba267348eed8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:00:12.233698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-896ac805-a745-4d25-bc24-2dcfa0384358 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:00:12.234358 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-896ac805-a745-4d25-bc24-2dcfa0384358 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:00:12.236061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1037/2034] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:00:12 2022] DELETE /identity/v3/projects/486aaad8b7324b38b38eba267348eed8 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:00:12.241499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe30aa1-e1dc-4910-955b-a4eab54fcca4 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:12.241813 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe30aa1-e1dc-4910-955b-a4eab54fcca4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:12.242095 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe30aa1-e1dc-4910-955b-a4eab54fcca4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:12.264567 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-bbe30aa1-e1dc-4910-955b-a4eab54fcca4 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:00:12.265839 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bbe30aa1-e1dc-4910-955b-a4eab54fcca4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:12.334422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 998/2035] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:00:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:00:12.344055 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4ec664-88a3-4b9a-9ec8-9c27fdc7b5f2 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:12.344346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4ec664-88a3-4b9a-9ec8-9c27fdc7b5f2 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:12.344767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4ec664-88a3-4b9a-9ec8-9c27fdc7b5f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:12.366553 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-4c4ec664-88a3-4b9a-9ec8-9c27fdc7b5f2 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:00:12.370915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4c4ec664-88a3-4b9a-9ec8-9c27fdc7b5f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:12.378426 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1038/2036] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:00:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 01:00:12.382926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe30aa1-e1dc-4910-955b-a4eab54fcca4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:12.398600 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe30aa1-e1dc-4910-955b-a4eab54fcca4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:12.405218 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3356c2-ae88-46f6-86e2-dedec2c6228e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:12.406519 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3356c2-ae88-46f6-86e2-dedec2c6228e admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:12.407025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3356c2-ae88-46f6-86e2-dedec2c6228e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:12.407537 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3356c2-ae88-46f6-86e2-dedec2c6228e admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:12.408538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3356c2-ae88-46f6-86e2-dedec2c6228e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:00:12.409010 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3356c2-ae88-46f6-86e2-dedec2c6228e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:12.410905 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3356c2-ae88-46f6-86e2-dedec2c6228e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:12.437561 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 999/2037] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:00:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:00:12.438195 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4ec664-88a3-4b9a-9ec8-9c27fdc7b5f2 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:12.450409 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4ec664-88a3-4b9a-9ec8-9c27fdc7b5f2 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:12.460541 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5cd58d-0a2d-4d6b-8262-787cf469e8c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:12.461357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5cd58d-0a2d-4d6b-8262-787cf469e8c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:12.461870 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5cd58d-0a2d-4d6b-8262-787cf469e8c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:12.462748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5cd58d-0a2d-4d6b-8262-787cf469e8c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:12.463802 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5cd58d-0a2d-4d6b-8262-787cf469e8c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:12.465875 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5cd58d-0a2d-4d6b-8262-787cf469e8c4 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:12.513009 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1039/2038] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:00:12 2022] POST /identity/v3/projects => generated 371 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 01:00:12.514654 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3356c2-ae88-46f6-86e2-dedec2c6228e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:12.527552 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3356c2-ae88-46f6-86e2-dedec2c6228e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:12.595114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf0d339-39ec-46b6-9f3d-09b6752fc0f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:12.595736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf0d339-39ec-46b6-9f3d-09b6752fc0f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:12.596013 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf0d339-39ec-46b6-9f3d-09b6752fc0f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:12.596284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf0d339-39ec-46b6-9f3d-09b6752fc0f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:12.596940 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf0d339-39ec-46b6-9f3d-09b6752fc0f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:12.598905 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf0d339-39ec-46b6-9f3d-09b6752fc0f5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:12.626937 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1000/2039] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:00:12 2022] POST /identity/v3/users => generated 286 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:00:12.630095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5cd58d-0a2d-4d6b-8262-787cf469e8c4 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:12.651802 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5cd58d-0a2d-4d6b-8262-787cf469e8c4 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:12.659863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad09246-1108-46fe-8f0f-9d33b194234e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:12.660499 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad09246-1108-46fe-8f0f-9d33b194234e admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:12.660788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad09246-1108-46fe-8f0f-9d33b194234e admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:12.661050 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad09246-1108-46fe-8f0f-9d33b194234e admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:12.661710 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad09246-1108-46fe-8f0f-9d33b194234e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:12.663547 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad09246-1108-46fe-8f0f-9d33b194234e admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:12.682001 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1040/2040] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:00:12 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:00:12.686490 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf0d339-39ec-46b6-9f3d-09b6752fc0f5 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:12.698196 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf0d339-39ec-46b6-9f3d-09b6752fc0f5 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:12.703691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0c430b-2396-429d-836c-038a58f84822 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:12.704300 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c430b-2396-429d-836c-038a58f84822 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:12.704576 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c430b-2396-429d-836c-038a58f84822 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:12.704845 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c430b-2396-429d-836c-038a58f84822 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:12.705482 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0c430b-2396-429d-836c-038a58f84822 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:12.707135 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0c430b-2396-429d-836c-038a58f84822 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:12.718637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1001/2041] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:00:12 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:00:12.723251 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad09246-1108-46fe-8f0f-9d33b194234e admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:12.734254 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad09246-1108-46fe-8f0f-9d33b194234e admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:12.738995 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b2011c-2047-47ad-b906-211278f1de64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:12.739657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b2011c-2047-47ad-b906-211278f1de64 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:12.739942 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b2011c-2047-47ad-b906-211278f1de64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:12.740218 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b2011c-2047-47ad-b906-211278f1de64 admin admin] PATH_INFO: `/v3/projects/ed24cab62e1f4810a10e7ba36fcbeadb/users/b899e194369a4efcb87d7cb92dc5f902/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:12.755429 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b2011c-2047-47ad-b906-211278f1de64 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ed24cab62e1f4810a10e7ba36fcbeadb, user_id=b899e194369a4efcb87d7cb92dc5f902, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:12.758370 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b2011c-2047-47ad-b906-211278f1de64 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:12.772579 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1041/2042] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:00:12 2022] PUT /identity/v3/projects/ed24cab62e1f4810a10e7ba36fcbeadb/users/b899e194369a4efcb87d7cb92dc5f902/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:00:12.778083 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8004625f-f4db-4026-93f2-42f1bee4a655 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:12.778604 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8004625f-f4db-4026-93f2-42f1bee4a655 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:12.779217 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8004625f-f4db-4026-93f2-42f1bee4a655 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:12.799367 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-8004625f-f4db-4026-93f2-42f1bee4a655 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:00:12.800914 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8004625f-f4db-4026-93f2-42f1bee4a655 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:12.810604 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:00:12.866961 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1002/2043] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:00:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:00:12.878510 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b2011c-2047-47ad-b906-211278f1de64 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:12.880254 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d7b2011c-2047-47ad-b906-211278f1de64 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:12.945419 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b2011c-2047-47ad-b906-211278f1de64 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:12.950746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7595499b-358f-47e4-96be-dba34138543f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:12.951539 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7595499b-358f-47e4-96be-dba34138543f service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:12.951831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7595499b-358f-47e4-96be-dba34138543f service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:12.952111 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7595499b-358f-47e4-96be-dba34138543f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:12.962647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7595499b-358f-47e4-96be-dba34138543f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:12.964145 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7595499b-358f-47e4-96be-dba34138543f service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:12.974648 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1042/2044] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:00:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:00:17.262226 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19754632-8687-42f5-97cd-727c21c3aaf7 None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:17.262693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19754632-8687-42f5-97cd-727c21c3aaf7 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:17.262980 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19754632-8687-42f5-97cd-727c21c3aaf7 None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:17.264874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1003/2045] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 16 01:00:17 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 01:00:17.274930 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b98d2d-f23b-4f62-b002-ada14363992d None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:17.275920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b98d2d-f23b-4f62-b002-ada14363992d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:17.276370 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b98d2d-f23b-4f62-b002-ada14363992d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:17.286962 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:00:17.316529 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-e7b98d2d-f23b-4f62-b002-ada14363992d None None] MFA Rules not processed for user `4eba26813e154595a12a67fe4be5eeff`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:00:17.319783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-e7b98d2d-f23b-4f62-b002-ada14363992d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:17.524932 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1043/2046] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 16 01:00:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:00:17.537770 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19754632-8687-42f5-97cd-727c21c3aaf7 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:17.540266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-19754632-8687-42f5-97cd-727c21c3aaf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:17.675683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19754632-8687-42f5-97cd-727c21c3aaf7 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:17.681728 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33449807-2426-457b-baf0-1bcd2dd46d41 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '78473c29cef449b89bc9185de67f1bf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:17.682723 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33449807-2426-457b-baf0-1bcd2dd46d41 service glance] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:17.683838 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33449807-2426-457b-baf0-1bcd2dd46d41 service glance] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:17.684365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33449807-2426-457b-baf0-1bcd2dd46d41 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:17.711198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33449807-2426-457b-baf0-1bcd2dd46d41 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:17.713599 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33449807-2426-457b-baf0-1bcd2dd46d41 service glance] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:17.741025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1004/2047] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:00:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:00:17.894841 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c12d35f-0743-4f0e-833a-a563050e8182 None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:17.895709 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c12d35f-0743-4f0e-833a-a563050e8182 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:17.896085 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c12d35f-0743-4f0e-833a-a563050e8182 None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:17.897885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1044/2048] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 16 01:00:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 01:00:17.904154 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1a6ce7-aada-4c57-890b-479c4d97ac35 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:17.904584 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1a6ce7-aada-4c57-890b-479c4d97ac35 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:17.904953 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1a6ce7-aada-4c57-890b-479c4d97ac35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:17.955343 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3c1a6ce7-aada-4c57-890b-479c4d97ac35 None None] MFA Rules not processed for user `f4eb1a640186411097cb1e3bb9e92020`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:00:17.956939 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3c1a6ce7-aada-4c57-890b-479c4d97ac35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:17.985711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1005/2049] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 16 01:00:17 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:00:18.032231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c12d35f-0743-4f0e-833a-a563050e8182 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:18.034477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-5c12d35f-0743-4f0e-833a-a563050e8182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:18.052836 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c12d35f-0743-4f0e-833a-a563050e8182 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:18.058195 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f60aa3-12b5-4f1f-91e0-ecc8aa0b9d88 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:18.059865 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f60aa3-12b5-4f1f-91e0-ecc8aa0b9d88 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:18.060423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f60aa3-12b5-4f1f-91e0-ecc8aa0b9d88 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:18.061082 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f60aa3-12b5-4f1f-91e0-ecc8aa0b9d88 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:18.070459 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f60aa3-12b5-4f1f-91e0-ecc8aa0b9d88 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:18.077586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f60aa3-12b5-4f1f-91e0-ecc8aa0b9d88 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:18.088998 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1045/2050] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:00:18 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:00:18.278971 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39969dce-9ce3-4f28-87a2-65149e76954f None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:18.279898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39969dce-9ce3-4f28-87a2-65149e76954f None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:18.280452 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39969dce-9ce3-4f28-87a2-65149e76954f None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:18.282178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1006/2051] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 16 01:00:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 01:00:18.288696 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0ac83e-337f-4a5e-80bb-d139922037f6 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:18.289204 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0ac83e-337f-4a5e-80bb-d139922037f6 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:18.289323 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0ac83e-337f-4a5e-80bb-d139922037f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:18.314172 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-7a0ac83e-337f-4a5e-80bb-d139922037f6 None None] MFA Rules not processed for user `4eba26813e154595a12a67fe4be5eeff`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:00:18.315906 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-7a0ac83e-337f-4a5e-80bb-d139922037f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:18.337631 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1046/2052] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 16 01:00:18 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:00:18.357450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39969dce-9ce3-4f28-87a2-65149e76954f None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:18.370086 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39969dce-9ce3-4f28-87a2-65149e76954f None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:18.375311 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46e3e2c-17e9-4bca-92e8-06c2dd8a7f99 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:18.375543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46e3e2c-17e9-4bca-92e8-06c2dd8a7f99 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:18.375821 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46e3e2c-17e9-4bca-92e8-06c2dd8a7f99 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:18.376098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46e3e2c-17e9-4bca-92e8-06c2dd8a7f99 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:18.395614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f46e3e2c-17e9-4bca-92e8-06c2dd8a7f99 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:18.398467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f46e3e2c-17e9-4bca-92e8-06c2dd8a7f99 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:18.419182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1007/2053] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:00:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:00:19.631497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e613e060-ca7c-4843-bb14-d79eefd501f8 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:19.632480 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e613e060-ca7c-4843-bb14-d79eefd501f8 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:19.632937 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e613e060-ca7c-4843-bb14-d79eefd501f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:19.674187 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-e613e060-ca7c-4843-bb14-d79eefd501f8 None None] MFA Rules not processed for user `b899e194369a4efcb87d7cb92dc5f902`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:00:19.675793 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-e613e060-ca7c-4843-bb14-d79eefd501f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:19.751590 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46e3e2c-17e9-4bca-92e8-06c2dd8a7f99 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:19.754520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f46e3e2c-17e9-4bca-92e8-06c2dd8a7f99 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:19.801873 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1047/2054] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:00:19 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:00:19.804274 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e613e060-ca7c-4843-bb14-d79eefd501f8 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:19.826688 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e613e060-ca7c-4843-bb14-d79eefd501f8 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:19.835383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab01395-4316-48c9-83fa-82da81935e9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:19.836255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab01395-4316-48c9-83fa-82da81935e9e admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:19.836980 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab01395-4316-48c9-83fa-82da81935e9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:19.837182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab01395-4316-48c9-83fa-82da81935e9e admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:19.838196 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab01395-4316-48c9-83fa-82da81935e9e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:19.841469 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab01395-4316-48c9-83fa-82da81935e9e admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:19.847723 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46e3e2c-17e9-4bca-92e8-06c2dd8a7f99 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:19.854552 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e57fd1-7800-4b91-9a5c-48f63fe43fda service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87785ccd37a642e08a0ad34a4fb1c754', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:19.855232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e57fd1-7800-4b91-9a5c-48f63fe43fda service placement] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:19.855556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e57fd1-7800-4b91-9a5c-48f63fe43fda service placement] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:19.855838 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e57fd1-7800-4b91-9a5c-48f63fe43fda service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:19.858063 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-91e57fd1-7800-4b91-9a5c-48f63fe43fda service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:19.872461 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e57fd1-7800-4b91-9a5c-48f63fe43fda service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:19.874400 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e57fd1-7800-4b91-9a5c-48f63fe43fda service placement] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:19.896012 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1048/2055] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:00:19 2022] POST /identity/v3/projects => generated 371 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:00:19.897504 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab01395-4316-48c9-83fa-82da81935e9e admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:19.899780 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1008/2056] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:00:19 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:00:19.972088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:00:19.999116 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab01395-4316-48c9-83fa-82da81935e9e admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:20.004708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a9c86b-1e71-4eb7-ad07-ce50970ad8bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:20.005211 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a9c86b-1e71-4eb7-ad07-ce50970ad8bd admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:20.005710 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a9c86b-1e71-4eb7-ad07-ce50970ad8bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:20.005999 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a9c86b-1e71-4eb7-ad07-ce50970ad8bd admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:20.006648 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a9c86b-1e71-4eb7-ad07-ce50970ad8bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:20.009292 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a9c86b-1e71-4eb7-ad07-ce50970ad8bd admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:20.046376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1049/2057] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:00:19 2022] POST /identity/v3/users => generated 292 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:00:20.050332 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e57fd1-7800-4b91-9a5c-48f63fe43fda service placement] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:20.065351 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e57fd1-7800-4b91-9a5c-48f63fe43fda service placement] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:20.072167 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49e0755-456d-43ff-8fe1-13d6f669c641 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:20.072849 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49e0755-456d-43ff-8fe1-13d6f669c641 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:20.073134 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49e0755-456d-43ff-8fe1-13d6f669c641 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:20.073409 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49e0755-456d-43ff-8fe1-13d6f669c641 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:20.074066 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49e0755-456d-43ff-8fe1-13d6f669c641 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:20.076831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49e0755-456d-43ff-8fe1-13d6f669c641 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:20.095661 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1009/2058] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:00:20 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 01:00:20.100593 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a9c86b-1e71-4eb7-ad07-ce50970ad8bd admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:20.117356 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a9c86b-1e71-4eb7-ad07-ce50970ad8bd admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:20.129174 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651a0e40-e7e1-4697-89d6-5ac72c30fd28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:20.129879 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651a0e40-e7e1-4697-89d6-5ac72c30fd28 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:20.130281 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651a0e40-e7e1-4697-89d6-5ac72c30fd28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:20.130479 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651a0e40-e7e1-4697-89d6-5ac72c30fd28 admin admin] PATH_INFO: `/v3/projects/ef2df25edb2b4bee9f71c94e2249e1f7/users/847949f2b2f84783bcad2fcd9177e811/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:20.166343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651a0e40-e7e1-4697-89d6-5ac72c30fd28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef2df25edb2b4bee9f71c94e2249e1f7, user_id=847949f2b2f84783bcad2fcd9177e811, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:20.173352 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651a0e40-e7e1-4697-89d6-5ac72c30fd28 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:20.194915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1050/2059] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:00:20 2022] PUT /identity/v3/projects/ef2df25edb2b4bee9f71c94e2249e1f7/users/847949f2b2f84783bcad2fcd9177e811/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:00:20.198272 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49e0755-456d-43ff-8fe1-13d6f669c641 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:20.210408 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49e0755-456d-43ff-8fe1-13d6f669c641 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:20.237688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:00:20.302436 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cd3b78-720e-48ed-a0b1-2fbe39e3fcd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:20.303200 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cd3b78-720e-48ed-a0b1-2fbe39e3fcd5 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:20.303546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cd3b78-720e-48ed-a0b1-2fbe39e3fcd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:20.303809 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cd3b78-720e-48ed-a0b1-2fbe39e3fcd5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:20.304471 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cd3b78-720e-48ed-a0b1-2fbe39e3fcd5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:20.306288 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cd3b78-720e-48ed-a0b1-2fbe39e3fcd5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:20.333370 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1010/2060] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:00:20 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:00:20.338408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651a0e40-e7e1-4697-89d6-5ac72c30fd28 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:20.350201 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651a0e40-e7e1-4697-89d6-5ac72c30fd28 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:20.355909 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6303a1-19d2-4dda-929e-46dd0c5f897a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:20.356710 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6303a1-19d2-4dda-929e-46dd0c5f897a admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:20.356996 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6303a1-19d2-4dda-929e-46dd0c5f897a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:20.357271 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6303a1-19d2-4dda-929e-46dd0c5f897a admin admin] PATH_INFO: `/v3/projects/ef2df25edb2b4bee9f71c94e2249e1f7/users/847949f2b2f84783bcad2fcd9177e811/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:20.359226 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6303a1-19d2-4dda-929e-46dd0c5f897a admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef2df25edb2b4bee9f71c94e2249e1f7, user_id=847949f2b2f84783bcad2fcd9177e811, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:20.361925 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6303a1-19d2-4dda-929e-46dd0c5f897a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:20.373787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1051/2061] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:00:20 2022] PUT /identity/v3/projects/ef2df25edb2b4bee9f71c94e2249e1f7/users/847949f2b2f84783bcad2fcd9177e811/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:00:23.882572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fb92f0-5fb8-4bd6-8c19-987e908600c0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:23.883750 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fb92f0-5fb8-4bd6-8c19-987e908600c0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:23.883750 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fb92f0-5fb8-4bd6-8c19-987e908600c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:23.917910 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-91fb92f0-5fb8-4bd6-8c19-987e908600c0 None None] MFA Rules not processed for user `33b136afbf8a4d98a9d62053b68707e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:00:23.922805 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-91fb92f0-5fb8-4bd6-8c19-987e908600c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:24.009767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1011/2062] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 16 01:00:23 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:00:24.021922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6303a1-19d2-4dda-929e-46dd0c5f897a admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:24.024221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-ff6303a1-19d2-4dda-929e-46dd0c5f897a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:24.129547 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6303a1-19d2-4dda-929e-46dd0c5f897a admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:24.146352 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33280f78-57bf-4cb6-9bec-2b16a0071689 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de64d498da78448394b0d4d1851a2789', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:24.147099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33280f78-57bf-4cb6-9bec-2b16a0071689 service swift] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:24.147467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33280f78-57bf-4cb6-9bec-2b16a0071689 service swift] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:24.147746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33280f78-57bf-4cb6-9bec-2b16a0071689 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:24.163575 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33280f78-57bf-4cb6-9bec-2b16a0071689 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:24.166310 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33280f78-57bf-4cb6-9bec-2b16a0071689 service swift] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:24.182559 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1052/2063] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 16 01:00:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 16 01:00:24.597871 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfa49f0-74c2-4c18-8af7-e005e64eb049 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:24.598280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfa49f0-74c2-4c18-8af7-e005e64eb049 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:24.598739 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfa49f0-74c2-4c18-8af7-e005e64eb049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:24.644121 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-4bfa49f0-74c2-4c18-8af7-e005e64eb049 None None] MFA Rules not processed for user `c4e65cef89af48a7a51e5548634de75f`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:00:24.646448 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4bfa49f0-74c2-4c18-8af7-e005e64eb049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:24.854886 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1012/2064] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:00:24 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:00:28.382301 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6702465e-cda9-44ee-ab31-611cb261ce84 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:28.385015 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6702465e-cda9-44ee-ab31-611cb261ce84 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:28.385288 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6702465e-cda9-44ee-ab31-611cb261ce84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:28.470649 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-6702465e-cda9-44ee-ab31-611cb261ce84 None None] MFA Rules not processed for user `847949f2b2f84783bcad2fcd9177e811`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:00:28.472112 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6702465e-cda9-44ee-ab31-611cb261ce84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:28.596277 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1053/2065] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:00:28 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:00:28.611182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfa49f0-74c2-4c18-8af7-e005e64eb049 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:28.612722 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4bfa49f0-74c2-4c18-8af7-e005e64eb049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:28.704461 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:00:28.714778 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfa49f0-74c2-4c18-8af7-e005e64eb049 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:28.720187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9e788b-13a3-45b7-9e5c-403bdab2672d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:28.720809 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9e788b-13a3-45b7-9e5c-403bdab2672d service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:28.721088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9e788b-13a3-45b7-9e5c-403bdab2672d service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:28.721356 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9e788b-13a3-45b7-9e5c-403bdab2672d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:28.780815 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9e788b-13a3-45b7-9e5c-403bdab2672d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:28.782445 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9e788b-13a3-45b7-9e5c-403bdab2672d service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:28.792148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1013/2066] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:00:28 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:00:30.489602 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc27b6e1-6793-4451-81fd-097404aa6c20 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:30.490608 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc27b6e1-6793-4451-81fd-097404aa6c20 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:30.491104 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc27b6e1-6793-4451-81fd-097404aa6c20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:30.527803 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-dc27b6e1-6793-4451-81fd-097404aa6c20 None None] MFA Rules not processed for user `b899e194369a4efcb87d7cb92dc5f902`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:00:30.529705 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-dc27b6e1-6793-4451-81fd-097404aa6c20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:30.539753 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1054/2067] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:00:30 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:00:38.787481 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=28 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:00:40.520301 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=93,keystone:INSERT=5 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:00:44.610787 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06304f52-d221-4c2d-855e-95c5c5053640 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:44.611359 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06304f52-d221-4c2d-855e-95c5c5053640 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:44.611359 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06304f52-d221-4c2d-855e-95c5c5053640 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:44.632182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-06304f52-d221-4c2d-855e-95c5c5053640 None None] MFA Rules not processed for user `33b136afbf8a4d98a9d62053b68707e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:00:44.633424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-06304f52-d221-4c2d-855e-95c5c5053640 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:44.642016 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1014/2068] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 16 01:00:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:00:44.655117 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc27b6e1-6793-4451-81fd-097404aa6c20 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:44.667616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc27b6e1-6793-4451-81fd-097404aa6c20 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:44.673099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f8e684-9c5d-4876-96ac-e6da4e3c9b57 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de64d498da78448394b0d4d1851a2789', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:44.673746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f8e684-9c5d-4876-96ac-e6da4e3c9b57 service swift] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:44.674039 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f8e684-9c5d-4876-96ac-e6da4e3c9b57 service swift] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:44.674314 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f8e684-9c5d-4876-96ac-e6da4e3c9b57 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:44.684655 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f8e684-9c5d-4876-96ac-e6da4e3c9b57 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:44.686316 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f8e684-9c5d-4876-96ac-e6da4e3c9b57 service swift] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:44.695972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1055/2069] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 16 01:00:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 16 01:00:54.626901 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:00:54.691671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=6 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:00:55.903026 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06304f52-d221-4c2d-855e-95c5c5053640 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:56.049493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06304f52-d221-4c2d-855e-95c5c5053640 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:56.057942 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968d527b-eae1-491b-9cb1-6aec003a2086 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:56.059039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968d527b-eae1-491b-9cb1-6aec003a2086 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:56.059531 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968d527b-eae1-491b-9cb1-6aec003a2086 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:56.060026 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968d527b-eae1-491b-9cb1-6aec003a2086 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:56.064653 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-968d527b-eae1-491b-9cb1-6aec003a2086 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:56.211399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968d527b-eae1-491b-9cb1-6aec003a2086 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:56.213478 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968d527b-eae1-491b-9cb1-6aec003a2086 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:56.228104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1015/2070] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:00:55 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:00:57.737028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25309a32-8aa8-4768-8367-d3fd53b6e592 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:57.738440 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25309a32-8aa8-4768-8367-d3fd53b6e592 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:57.738984 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25309a32-8aa8-4768-8367-d3fd53b6e592 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:57.761744 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-25309a32-8aa8-4768-8367-d3fd53b6e592 None None] MFA Rules not processed for user `33b136afbf8a4d98a9d62053b68707e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:00:57.763436 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-25309a32-8aa8-4768-8367-d3fd53b6e592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:00:57.779588 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1056/2071] 10.222.0.25 () {58 vars in 967 bytes} [Sun Oct 16 01:00:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:00:57.844083 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968d527b-eae1-491b-9cb1-6aec003a2086 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:00:57.860588 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968d527b-eae1-491b-9cb1-6aec003a2086 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:00:57.867570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931f8c88-216f-406f-a19c-94f9697a790e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de64d498da78448394b0d4d1851a2789', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:00:57.868492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931f8c88-216f-406f-a19c-94f9697a790e service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:00:57.869450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931f8c88-216f-406f-a19c-94f9697a790e service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:00:57.869886 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931f8c88-216f-406f-a19c-94f9697a790e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:00:57.889629 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931f8c88-216f-406f-a19c-94f9697a790e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:00:57.892246 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931f8c88-216f-406f-a19c-94f9697a790e service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:00:57.911104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1016/2072] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 16 01:00:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 16 01:01:05.195586 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3843d95-0448-4d09-a0a1-a8e117bfd563 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:05.196223 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3843d95-0448-4d09-a0a1-a8e117bfd563 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:05.196223 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3843d95-0448-4d09-a0a1-a8e117bfd563 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:05.215881 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-b3843d95-0448-4d09-a0a1-a8e117bfd563 None None] MFA Rules not processed for user `33b136afbf8a4d98a9d62053b68707e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:01:05.217193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b3843d95-0448-4d09-a0a1-a8e117bfd563 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:05.224853 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1057/2073] 10.222.0.25 () {58 vars in 967 bytes} [Sun Oct 16 01:01:05 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:01:05.238000 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931f8c88-216f-406f-a19c-94f9697a790e service swift] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:05.249260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931f8c88-216f-406f-a19c-94f9697a790e service swift] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:05.254361 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9a87b5-978d-4ca6-9834-331e657d8f8d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de64d498da78448394b0d4d1851a2789', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:05.255195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a87b5-978d-4ca6-9834-331e657d8f8d service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:05.255350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a87b5-978d-4ca6-9834-331e657d8f8d service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:05.255540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a87b5-978d-4ca6-9834-331e657d8f8d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:05.264066 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9a87b5-978d-4ca6-9834-331e657d8f8d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:05.265658 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9a87b5-978d-4ca6-9834-331e657d8f8d service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:05.273842 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1017/2074] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 16 01:01:05 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 16 01:01:07.297046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3843d95-0448-4d09-a0a1-a8e117bfd563 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:07.299449 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b3843d95-0448-4d09-a0a1-a8e117bfd563 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:07.378034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3843d95-0448-4d09-a0a1-a8e117bfd563 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:07.382853 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89733ddf-f36b-43c3-8807-b811a92f3ea0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:07.383609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89733ddf-f36b-43c3-8807-b811a92f3ea0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:07.383963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89733ddf-f36b-43c3-8807-b811a92f3ea0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:07.384337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89733ddf-f36b-43c3-8807-b811a92f3ea0 admin admin] PATH_INFO: `/v3/users/e04650eb7a9349e3b4f28ded86d26b43` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:07.385696 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89733ddf-f36b-43c3-8807-b811a92f3ea0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e04650eb7a9349e3b4f28ded86d26b43)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:07.387835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89733ddf-f36b-43c3-8807-b811a92f3ea0 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:07.457874 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-89733ddf-f36b-43c3-8807-b811a92f3ea0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e04650eb7a9349e3b4f28ded86d26b43'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:01:07.458626 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-89733ddf-f36b-43c3-8807-b811a92f3ea0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e04650eb7a9349e3b4f28ded86d26b43'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:01:07.468424 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-89733ddf-f36b-43c3-8807-b811a92f3ea0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e04650eb7a9349e3b4f28ded86d26b43'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:01:07.484533 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-89733ddf-f36b-43c3-8807-b811a92f3ea0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e04650eb7a9349e3b4f28ded86d26b43'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:01:07.494188 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-89733ddf-f36b-43c3-8807-b811a92f3ea0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e04650eb7a9349e3b4f28ded86d26b43'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:01:07.503059 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1058/2075] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:01:07 2022] DELETE /identity/v3/users/e04650eb7a9349e3b4f28ded86d26b43 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:01:07.689564 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31226f2-4adf-470f-9253-87aa0a252363 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:07.690169 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31226f2-4adf-470f-9253-87aa0a252363 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:07.690258 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31226f2-4adf-470f-9253-87aa0a252363 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:07.700918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89733ddf-f36b-43c3-8807-b811a92f3ea0 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:07.702373 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-89733ddf-f36b-43c3-8807-b811a92f3ea0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:07.737523 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a31226f2-4adf-470f-9253-87aa0a252363 None None] MFA Rules not processed for user `466f2f8ddd184a98bdb5e922818905cc`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:01:07.739548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a31226f2-4adf-470f-9253-87aa0a252363 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:07.802491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89733ddf-f36b-43c3-8807-b811a92f3ea0 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:07.808440 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b77e61-2494-4c3b-9055-c26a60ee114e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:07.811043 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b77e61-2494-4c3b-9055-c26a60ee114e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:07.811676 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b77e61-2494-4c3b-9055-c26a60ee114e admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:07.813125 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b77e61-2494-4c3b-9055-c26a60ee114e admin admin] PATH_INFO: `/v3/projects/92116166994646b39ec7a4c23e8cedc4` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:07.815886 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b77e61-2494-4c3b-9055-c26a60ee114e admin admin] RBAC: Authorizing `identity:delete_project(project_id=92116166994646b39ec7a4c23e8cedc4)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:07.817694 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b77e61-2494-4c3b-9055-c26a60ee114e admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:07.832571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1018/2076] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:01:07 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:01:07.839046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:01:07.885829 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c1b77e61-2494-4c3b-9055-c26a60ee114e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '92116166994646b39ec7a4c23e8cedc4', 'request_id': 'req-c1b77e61-2494-4c3b-9055-c26a60ee114e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:01:07.901802 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c1b77e61-2494-4c3b-9055-c26a60ee114e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '92116166994646b39ec7a4c23e8cedc4', 'request_id': 'req-c1b77e61-2494-4c3b-9055-c26a60ee114e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:01:07.920090 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c1b77e61-2494-4c3b-9055-c26a60ee114e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '92116166994646b39ec7a4c23e8cedc4', 'request_id': 'req-c1b77e61-2494-4c3b-9055-c26a60ee114e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:01:07.933169 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c1b77e61-2494-4c3b-9055-c26a60ee114e admin admin] The token cache is being invalidate because project 92116166994646b39ec7a4c23e8cedc4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:01:07.933462 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c1b77e61-2494-4c3b-9055-c26a60ee114e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:01:07.934138 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c1b77e61-2494-4c3b-9055-c26a60ee114e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:01:07.935816 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1059/2077] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:01:07 2022] DELETE /identity/v3/projects/92116166994646b39ec7a4c23e8cedc4 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:01:07.941115 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd225537-460e-4e68-9272-113b43a76ece None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:07.941440 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd225537-460e-4e68-9272-113b43a76ece None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:07.941720 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd225537-460e-4e68-9272-113b43a76ece None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:07.964573 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-cd225537-460e-4e68-9272-113b43a76ece None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:01:07.966121 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cd225537-460e-4e68-9272-113b43a76ece None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:07.991123 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:01:08.038314 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1019/2078] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:01:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 01:01:08.042183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc41bb89-eca2-404d-aa39-f177248bbb16 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:08.042545 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc41bb89-eca2-404d-aa39-f177248bbb16 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:08.042852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc41bb89-eca2-404d-aa39-f177248bbb16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:08.065709 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-dc41bb89-eca2-404d-aa39-f177248bbb16 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:01:08.067042 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-dc41bb89-eca2-404d-aa39-f177248bbb16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:08.078949 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1060/2079] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:01:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:01:08.081544 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd225537-460e-4e68-9272-113b43a76ece None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:08.091873 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd225537-460e-4e68-9272-113b43a76ece None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:08.097760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93f4127-2a5a-490a-8964-19a5ede6d745 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:08.098506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93f4127-2a5a-490a-8964-19a5ede6d745 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:08.098927 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93f4127-2a5a-490a-8964-19a5ede6d745 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:08.099308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93f4127-2a5a-490a-8964-19a5ede6d745 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:08.100073 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93f4127-2a5a-490a-8964-19a5ede6d745 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:01:08.100485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93f4127-2a5a-490a-8964-19a5ede6d745 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:08.102889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93f4127-2a5a-490a-8964-19a5ede6d745 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:08.115417 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1020/2080] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:01:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:01:08.120546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc41bb89-eca2-404d-aa39-f177248bbb16 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:08.130014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc41bb89-eca2-404d-aa39-f177248bbb16 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:08.134964 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4785f57d-2c79-4b9d-a6f0-b0a9e0e8d33c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:08.135596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4785f57d-2c79-4b9d-a6f0-b0a9e0e8d33c admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:08.135881 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4785f57d-2c79-4b9d-a6f0-b0a9e0e8d33c admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:08.136185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4785f57d-2c79-4b9d-a6f0-b0a9e0e8d33c admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:08.136807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4785f57d-2c79-4b9d-a6f0-b0a9e0e8d33c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:08.138792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4785f57d-2c79-4b9d-a6f0-b0a9e0e8d33c admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:08.171318 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1061/2081] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:01:08 2022] POST /identity/v3/projects => generated 385 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:01:08.172767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93f4127-2a5a-490a-8964-19a5ede6d745 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:08.184120 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93f4127-2a5a-490a-8964-19a5ede6d745 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:08.248085 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745ad16a-b46c-4c3c-b936-50694a109410 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:08.248708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745ad16a-b46c-4c3c-b936-50694a109410 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:08.248993 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745ad16a-b46c-4c3c-b936-50694a109410 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:08.249269 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745ad16a-b46c-4c3c-b936-50694a109410 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:08.249909 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745ad16a-b46c-4c3c-b936-50694a109410 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:08.251893 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745ad16a-b46c-4c3c-b936-50694a109410 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:08.275068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1021/2082] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:01:08 2022] POST /identity/v3/users => generated 293 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 01:01:08.279265 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4785f57d-2c79-4b9d-a6f0-b0a9e0e8d33c admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:08.290270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4785f57d-2c79-4b9d-a6f0-b0a9e0e8d33c admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:08.294953 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd50a6d7-03a6-4368-8857-fceb9032e9fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:08.295571 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd50a6d7-03a6-4368-8857-fceb9032e9fe admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:08.295853 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd50a6d7-03a6-4368-8857-fceb9032e9fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:08.296146 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd50a6d7-03a6-4368-8857-fceb9032e9fe admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:08.296843 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd50a6d7-03a6-4368-8857-fceb9032e9fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:08.298561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd50a6d7-03a6-4368-8857-fceb9032e9fe admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:08.310612 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1062/2083] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:01:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:01:08.314524 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745ad16a-b46c-4c3c-b936-50694a109410 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:08.325331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745ad16a-b46c-4c3c-b936-50694a109410 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:08.329981 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbf3841-d2bf-47f8-bc76-a62fcab9ff95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:08.330594 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbf3841-d2bf-47f8-bc76-a62fcab9ff95 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:08.330903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbf3841-d2bf-47f8-bc76-a62fcab9ff95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:08.331184 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbf3841-d2bf-47f8-bc76-a62fcab9ff95 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:08.331836 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbf3841-d2bf-47f8-bc76-a62fcab9ff95 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:08.333465 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbf3841-d2bf-47f8-bc76-a62fcab9ff95 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:08.346845 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1022/2084] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:01:08 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:01:08.351080 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd50a6d7-03a6-4368-8857-fceb9032e9fe admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:08.362309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd50a6d7-03a6-4368-8857-fceb9032e9fe admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:08.367631 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536d4efc-fb6c-4116-9a39-8919f1b14169 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:08.368294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536d4efc-fb6c-4116-9a39-8919f1b14169 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:08.368571 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536d4efc-fb6c-4116-9a39-8919f1b14169 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:08.368844 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536d4efc-fb6c-4116-9a39-8919f1b14169 admin admin] PATH_INFO: `/v3/projects/343c019304314c669a3381cc4c68e481/users/a49486f43ff942bcbf4ef782d6eb6238/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:08.385028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536d4efc-fb6c-4116-9a39-8919f1b14169 admin admin] RBAC: Authorizing `identity:create_grant(project_id=343c019304314c669a3381cc4c68e481, user_id=a49486f43ff942bcbf4ef782d6eb6238, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:08.387808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536d4efc-fb6c-4116-9a39-8919f1b14169 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:08.401452 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1063/2085] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:01:08 2022] PUT /identity/v3/projects/343c019304314c669a3381cc4c68e481/users/a49486f43ff942bcbf4ef782d6eb6238/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:01:08.406984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158d3684-c677-4979-ab08-4456b4a57eb4 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:08.406984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158d3684-c677-4979-ab08-4456b4a57eb4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:08.407368 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158d3684-c677-4979-ab08-4456b4a57eb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:08.428395 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-158d3684-c677-4979-ab08-4456b4a57eb4 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:01:08.429691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-158d3684-c677-4979-ab08-4456b4a57eb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:08.495623 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1023/2086] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:01:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 01:01:08.502437 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536d4efc-fb6c-4116-9a39-8919f1b14169 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:08.504458 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-536d4efc-fb6c-4116-9a39-8919f1b14169 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:08.575996 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536d4efc-fb6c-4116-9a39-8919f1b14169 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:08.580660 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d375713-d33e-43c6-a86c-059a44dad33e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:08.581378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d375713-d33e-43c6-a86c-059a44dad33e service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:08.581862 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d375713-d33e-43c6-a86c-059a44dad33e service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:08.582205 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d375713-d33e-43c6-a86c-059a44dad33e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:08.592348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d375713-d33e-43c6-a86c-059a44dad33e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:08.594689 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d375713-d33e-43c6-a86c-059a44dad33e service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:08.609711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1064/2087] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:01:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:01:10.433934 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158d3684-c677-4979-ab08-4456b4a57eb4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:10.436776 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-158d3684-c677-4979-ab08-4456b4a57eb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:10.454404 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158d3684-c677-4979-ab08-4456b4a57eb4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:10.458284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d71b98-4b43-47a3-8eb8-a6164f3c45b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:10.458942 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d71b98-4b43-47a3-8eb8-a6164f3c45b0 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:10.459227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d71b98-4b43-47a3-8eb8-a6164f3c45b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:10.459506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d71b98-4b43-47a3-8eb8-a6164f3c45b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:10.461975 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f5d71b98-4b43-47a3-8eb8-a6164f3c45b0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:10.519744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:01:10.568024 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d71b98-4b43-47a3-8eb8-a6164f3c45b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:10.573493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d71b98-4b43-47a3-8eb8-a6164f3c45b0 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:10.588589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1024/2088] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:01:10 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:01:15.122025 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b12255-d91f-4c6d-9812-53f5a012d137 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:15.123547 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b12255-d91f-4c6d-9812-53f5a012d137 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:15.124112 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b12255-d91f-4c6d-9812-53f5a012d137 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:15.161653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-18b12255-d91f-4c6d-9812-53f5a012d137 None None] MFA Rules not processed for user `a49486f43ff942bcbf4ef782d6eb6238`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:01:15.163025 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-18b12255-d91f-4c6d-9812-53f5a012d137 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:15.194742 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:01:15.247174 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1065/2089] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:01:15 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:01:15.251525 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d71b98-4b43-47a3-8eb8-a6164f3c45b0 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:15.263568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d71b98-4b43-47a3-8eb8-a6164f3c45b0 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:15.269488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471a5e9a-c779-43b4-8f9b-b4b96aa45466 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:15.270456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471a5e9a-c779-43b4-8f9b-b4b96aa45466 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:15.271010 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471a5e9a-c779-43b4-8f9b-b4b96aa45466 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:15.271378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471a5e9a-c779-43b4-8f9b-b4b96aa45466 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:15.272139 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471a5e9a-c779-43b4-8f9b-b4b96aa45466 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:15.275001 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471a5e9a-c779-43b4-8f9b-b4b96aa45466 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:15.308216 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1025/2090] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:01:15 2022] POST /identity/v3/projects => generated 385 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:01:15.312350 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b12255-d91f-4c6d-9812-53f5a012d137 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:15.329973 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b12255-d91f-4c6d-9812-53f5a012d137 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:15.436066 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422e6bc1-ee11-436b-8ca4-fbe9be190e0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:15.436832 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422e6bc1-ee11-436b-8ca4-fbe9be190e0b admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:15.437218 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422e6bc1-ee11-436b-8ca4-fbe9be190e0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:15.437596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422e6bc1-ee11-436b-8ca4-fbe9be190e0b admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:15.438596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422e6bc1-ee11-436b-8ca4-fbe9be190e0b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:15.440789 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422e6bc1-ee11-436b-8ca4-fbe9be190e0b admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:15.472390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1066/2091] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:01:15 2022] POST /identity/v3/users => generated 299 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:01:15.476510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471a5e9a-c779-43b4-8f9b-b4b96aa45466 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:15.490137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471a5e9a-c779-43b4-8f9b-b4b96aa45466 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:15.498694 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6125ee9-c00f-44f6-ab2f-24d1d154d61f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:15.499798 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6125ee9-c00f-44f6-ab2f-24d1d154d61f admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:15.500346 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6125ee9-c00f-44f6-ab2f-24d1d154d61f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:15.501592 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6125ee9-c00f-44f6-ab2f-24d1d154d61f admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:15.502571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6125ee9-c00f-44f6-ab2f-24d1d154d61f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:15.505156 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6125ee9-c00f-44f6-ab2f-24d1d154d61f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:15.539720 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1026/2092] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:01:15 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:01:15.544515 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422e6bc1-ee11-436b-8ca4-fbe9be190e0b admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:15.568770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422e6bc1-ee11-436b-8ca4-fbe9be190e0b admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:15.579704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0b9c84-23ae-49c8-9aeb-7f3375881203 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:15.580510 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b9c84-23ae-49c8-9aeb-7f3375881203 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:15.580800 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b9c84-23ae-49c8-9aeb-7f3375881203 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:15.581078 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b9c84-23ae-49c8-9aeb-7f3375881203 admin admin] PATH_INFO: `/v3/projects/07fd28660dad4acf93dc9785e33d2366/users/9a16b4d57e56464e91a8cb5b1fb8950c/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:15.605623 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0b9c84-23ae-49c8-9aeb-7f3375881203 admin admin] RBAC: Authorizing `identity:create_grant(project_id=07fd28660dad4acf93dc9785e33d2366, user_id=9a16b4d57e56464e91a8cb5b1fb8950c, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:15.610376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0b9c84-23ae-49c8-9aeb-7f3375881203 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:15.628526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1067/2093] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:01:15 2022] PUT /identity/v3/projects/07fd28660dad4acf93dc9785e33d2366/users/9a16b4d57e56464e91a8cb5b1fb8950c/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:01:15.633373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6125ee9-c00f-44f6-ab2f-24d1d154d61f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:15.651571 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6125ee9-c00f-44f6-ab2f-24d1d154d61f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:15.740040 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c283a327-420c-4515-a7dc-3b2f7502af37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:15.740823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c283a327-420c-4515-a7dc-3b2f7502af37 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:15.741480 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c283a327-420c-4515-a7dc-3b2f7502af37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:15.741903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c283a327-420c-4515-a7dc-3b2f7502af37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:15.742834 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c283a327-420c-4515-a7dc-3b2f7502af37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:15.744594 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c283a327-420c-4515-a7dc-3b2f7502af37 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:15.757572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1027/2094] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:01:15 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:01:15.761771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0b9c84-23ae-49c8-9aeb-7f3375881203 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:15.776869 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0b9c84-23ae-49c8-9aeb-7f3375881203 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:15.782956 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4920f061-f712-424d-b745-a7ee13a6753d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:15.783629 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4920f061-f712-424d-b745-a7ee13a6753d admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:15.783908 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4920f061-f712-424d-b745-a7ee13a6753d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:15.784211 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4920f061-f712-424d-b745-a7ee13a6753d admin admin] PATH_INFO: `/v3/projects/07fd28660dad4acf93dc9785e33d2366/users/9a16b4d57e56464e91a8cb5b1fb8950c/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:15.786185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4920f061-f712-424d-b745-a7ee13a6753d admin admin] RBAC: Authorizing `identity:create_grant(project_id=07fd28660dad4acf93dc9785e33d2366, user_id=9a16b4d57e56464e91a8cb5b1fb8950c, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:15.789765 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4920f061-f712-424d-b745-a7ee13a6753d admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:15.800957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1068/2095] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:01:15 2022] PUT /identity/v3/projects/07fd28660dad4acf93dc9785e33d2366/users/9a16b4d57e56464e91a8cb5b1fb8950c/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:01:21.031769 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5213c17d-ae32-411f-adac-211877a3ab5d None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:21.033198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5213c17d-ae32-411f-adac-211877a3ab5d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:21.033673 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5213c17d-ae32-411f-adac-211877a3ab5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:21.078766 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-5213c17d-ae32-411f-adac-211877a3ab5d None None] MFA Rules not processed for user `9a16b4d57e56464e91a8cb5b1fb8950c`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:01:21.080112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5213c17d-ae32-411f-adac-211877a3ab5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:21.191530 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1028/2096] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:01:21 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:01:21.203126 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4920f061-f712-424d-b745-a7ee13a6753d admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:21.205206 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4920f061-f712-424d-b745-a7ee13a6753d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:21.316036 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4920f061-f712-424d-b745-a7ee13a6753d admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:21.321059 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74071f92-6a82-4316-9306-8096a0e0d2d4 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '78473c29cef449b89bc9185de67f1bf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:21.321891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74071f92-6a82-4316-9306-8096a0e0d2d4 service glance] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:21.322289 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74071f92-6a82-4316-9306-8096a0e0d2d4 service glance] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:21.322807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74071f92-6a82-4316-9306-8096a0e0d2d4 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:21.371147 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:01:21.410379 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74071f92-6a82-4316-9306-8096a0e0d2d4 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:21.412803 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74071f92-6a82-4316-9306-8096a0e0d2d4 service glance] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:21.427466 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1069/2097] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:01:21 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:01:21.624877 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b171aa-b305-49aa-9666-1c9a2bb82b79 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:21.625267 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b171aa-b305-49aa-9666-1c9a2bb82b79 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:21.625618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b171aa-b305-49aa-9666-1c9a2bb82b79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:21.652358 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-50b171aa-b305-49aa-9666-1c9a2bb82b79 None None] MFA Rules not processed for user `33b136afbf8a4d98a9d62053b68707e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:01:21.654021 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-50b171aa-b305-49aa-9666-1c9a2bb82b79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:21.668926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:01:21.730449 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1029/2098] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 16 01:01:21 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 01:01:21.745849 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74071f92-6a82-4316-9306-8096a0e0d2d4 service glance] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:21.748792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-74071f92-6a82-4316-9306-8096a0e0d2d4 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:21.832751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74071f92-6a82-4316-9306-8096a0e0d2d4 service glance] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:21.838352 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a7f0f5-a2b3-427c-8c28-e11169a804c8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de64d498da78448394b0d4d1851a2789', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:21.839163 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a7f0f5-a2b3-427c-8c28-e11169a804c8 service swift] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:21.839444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a7f0f5-a2b3-427c-8c28-e11169a804c8 service swift] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:21.839710 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a7f0f5-a2b3-427c-8c28-e11169a804c8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:21.850653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a7f0f5-a2b3-427c-8c28-e11169a804c8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:21.852931 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a7f0f5-a2b3-427c-8c28-e11169a804c8 service swift] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:21.862002 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1070/2099] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 16 01:01:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 16 01:01:31.438407 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d996abc6-adea-4c3a-9358-4f60ef13072b None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:31.438971 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d996abc6-adea-4c3a-9358-4f60ef13072b None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:31.438971 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d996abc6-adea-4c3a-9358-4f60ef13072b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:31.458253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-d996abc6-adea-4c3a-9358-4f60ef13072b None None] MFA Rules not processed for user `a49486f43ff942bcbf4ef782d6eb6238`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:01:31.460018 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-d996abc6-adea-4c3a-9358-4f60ef13072b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:31.466924 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1030/2100] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:01:31 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:01:31.858051 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=45 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:01:33.581930 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de717589-ec05-476b-9948-2b5d08cf574c None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:33.582532 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de717589-ec05-476b-9948-2b5d08cf574c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:33.582532 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de717589-ec05-476b-9948-2b5d08cf574c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:33.603533 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-de717589-ec05-476b-9948-2b5d08cf574c None None] MFA Rules not processed for user `33b136afbf8a4d98a9d62053b68707e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:01:33.604866 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-de717589-ec05-476b-9948-2b5d08cf574c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:33.612141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1071/2101] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 16 01:01:33 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:01:33.624903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d996abc6-adea-4c3a-9358-4f60ef13072b None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:01:33.637284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d996abc6-adea-4c3a-9358-4f60ef13072b None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:01:33.642306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cf423f-0155-4f40-b509-cf50f20598ac service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de64d498da78448394b0d4d1851a2789', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:01:33.643144 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cf423f-0155-4f40-b509-cf50f20598ac service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:33.643429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cf423f-0155-4f40-b509-cf50f20598ac service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:33.643713 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cf423f-0155-4f40-b509-cf50f20598ac service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:33.652677 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cf423f-0155-4f40-b509-cf50f20598ac service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:01:33.654314 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cf423f-0155-4f40-b509-cf50f20598ac service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:01:33.664201 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1031/2102] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 16 01:01:33 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 16 01:01:43.596929 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:01:43.660098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=26 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:01:50.170467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dec4d2-e148-458b-ab96-1bdcbe53c43c None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:50.171072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dec4d2-e148-458b-ab96-1bdcbe53c43c None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:50.171072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dec4d2-e148-458b-ab96-1bdcbe53c43c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:50.191025 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-65dec4d2-e148-458b-ab96-1bdcbe53c43c None None] MFA Rules not processed for user `b01ce2d27bdb4c91a3f55c891f949e00`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:01:50.192259 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-65dec4d2-e148-458b-ab96-1bdcbe53c43c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:50.263083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1072/2103] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:01:50 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:01:54.776702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e7d235-2561-4f54-9b53-fcc3aa5fa44d None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:01:54.777243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e7d235-2561-4f54-9b53-fcc3aa5fa44d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:01:54.777243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e7d235-2561-4f54-9b53-fcc3aa5fa44d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:01:54.797359 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-02e7d235-2561-4f54-9b53-fcc3aa5fa44d None None] MFA Rules not processed for user `ccece85a85fb48c4887b52c96c5bc642`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:01:54.798630 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-02e7d235-2561-4f54-9b53-fcc3aa5fa44d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:01:54.864064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1032/2104] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:01:54 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:02:00.252635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=25 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:02:04.851280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=25 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:02:06.786359 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dec4d2-e148-458b-ab96-1bdcbe53c43c None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:02:06.847682 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dec4d2-e148-458b-ab96-1bdcbe53c43c None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:02:06.851746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3417a93-efa2-431d-a186-423894893053 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:02:06.852360 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3417a93-efa2-431d-a186-423894893053 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:02:06.852636 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3417a93-efa2-431d-a186-423894893053 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:02:06.852906 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3417a93-efa2-431d-a186-423894893053 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:02:06.862571 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3417a93-efa2-431d-a186-423894893053 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:02:06.864170 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3417a93-efa2-431d-a186-423894893053 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:02:06.875451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1073/2105] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:02:06 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:02:16.870885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=26 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:02:18.245744 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e7d235-2561-4f54-9b53-fcc3aa5fa44d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:02:18.247246 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-02e7d235-2561-4f54-9b53-fcc3aa5fa44d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:02:18.306282 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e7d235-2561-4f54-9b53-fcc3aa5fa44d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:02:18.309927 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b976774-70af-4b54-80f1-a32c00909fb4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:02:18.310538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b976774-70af-4b54-80f1-a32c00909fb4 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:02:18.310816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b976774-70af-4b54-80f1-a32c00909fb4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:02:18.311127 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b976774-70af-4b54-80f1-a32c00909fb4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:02:18.313014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0b976774-70af-4b54-80f1-a32c00909fb4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:02:18.388459 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b976774-70af-4b54-80f1-a32c00909fb4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:02:18.389932 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b976774-70af-4b54-80f1-a32c00909fb4 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:02:18.402151 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1033/2106] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:02:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:02:18.576735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3417a93-efa2-431d-a186-423894893053 service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:02:18.586709 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3417a93-efa2-431d-a186-423894893053 service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:02:18.590381 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bc23de-8004-4948-ae8c-45038156055e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:02:18.591023 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bc23de-8004-4948-ae8c-45038156055e service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:02:18.591301 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bc23de-8004-4948-ae8c-45038156055e service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:02:18.591576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bc23de-8004-4948-ae8c-45038156055e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:02:18.593309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-12bc23de-8004-4948-ae8c-45038156055e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:02:18.607145 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bc23de-8004-4948-ae8c-45038156055e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:02:18.608581 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bc23de-8004-4948-ae8c-45038156055e service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:02:18.621299 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1074/2107] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:02:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:02:26.738578 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641bdfae-f3bf-498e-a790-b8363890000a None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:02:26.739106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641bdfae-f3bf-498e-a790-b8363890000a None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:02:26.739204 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641bdfae-f3bf-498e-a790-b8363890000a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:02:26.759079 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-641bdfae-f3bf-498e-a790-b8363890000a None None] MFA Rules not processed for user `466f2f8ddd184a98bdb5e922818905cc`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:02:26.760273 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-641bdfae-f3bf-498e-a790-b8363890000a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:02:26.815308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1034/2108] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:02:26 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:02:28.614849 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=6 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:02:34.935149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bc23de-8004-4948-ae8c-45038156055e service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:02:34.948423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bc23de-8004-4948-ae8c-45038156055e service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:02:34.952133 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88728edb-d91b-488e-8115-e791b20f650a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:02:34.952745 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88728edb-d91b-488e-8115-e791b20f650a service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:02:34.953067 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88728edb-d91b-488e-8115-e791b20f650a service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:02:34.953368 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88728edb-d91b-488e-8115-e791b20f650a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:02:34.955745 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-88728edb-d91b-488e-8115-e791b20f650a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:02:34.968137 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88728edb-d91b-488e-8115-e791b20f650a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:02:34.969587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88728edb-d91b-488e-8115-e791b20f650a service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:02:34.979296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1075/2109] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:02:34 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:02:36.781086 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641bdfae-f3bf-498e-a790-b8363890000a None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:02:36.782471 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-641bdfae-f3bf-498e-a790-b8363890000a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:02:36.799210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641bdfae-f3bf-498e-a790-b8363890000a None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:02:36.805294 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb59a056-e487-44e6-99ac-f98bc4ac14ec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:02:36.805942 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb59a056-e487-44e6-99ac-f98bc4ac14ec service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:02:36.806225 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb59a056-e487-44e6-99ac-f98bc4ac14ec service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:02:36.806516 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb59a056-e487-44e6-99ac-f98bc4ac14ec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:02:36.808540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bb59a056-e487-44e6-99ac-f98bc4ac14ec service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:02:36.820201 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb59a056-e487-44e6-99ac-f98bc4ac14ec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:02:36.822117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb59a056-e487-44e6-99ac-f98bc4ac14ec service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:02:36.832069 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1035/2110] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:02:36 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:02:44.974741 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=6 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:02:46.827679 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=85 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:02:51.336371 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88728edb-d91b-488e-8115-e791b20f650a service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:02:51.338064 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-88728edb-d91b-488e-8115-e791b20f650a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:02:51.353557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88728edb-d91b-488e-8115-e791b20f650a service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:02:51.358978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd2e464-84a7-47b1-a125-12c65435abeb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:02:51.359571 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd2e464-84a7-47b1-a125-12c65435abeb service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:02:51.359850 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd2e464-84a7-47b1-a125-12c65435abeb service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:02:51.360121 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd2e464-84a7-47b1-a125-12c65435abeb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:02:51.361792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-bbd2e464-84a7-47b1-a125-12c65435abeb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:02:51.372677 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd2e464-84a7-47b1-a125-12c65435abeb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:02:51.374248 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd2e464-84a7-47b1-a125-12c65435abeb service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:02:51.383151 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1076/2111] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:02:51 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:02:56.035481 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb59a056-e487-44e6-99ac-f98bc4ac14ec service nova] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:02:56.048687 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb59a056-e487-44e6-99ac-f98bc4ac14ec service nova] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:02:56.058893 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f272ff-bbe3-4af3-a57d-8044da225107 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:02:56.059956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f272ff-bbe3-4af3-a57d-8044da225107 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:02:56.060439 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f272ff-bbe3-4af3-a57d-8044da225107 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:02:56.060712 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f272ff-bbe3-4af3-a57d-8044da225107 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:02:56.063190 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f2f272ff-bbe3-4af3-a57d-8044da225107 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:02:56.080723 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f272ff-bbe3-4af3-a57d-8044da225107 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:02:56.082470 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f272ff-bbe3-4af3-a57d-8044da225107 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:02:56.094855 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1036/2112] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:02:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 01:02:56.935908 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd2e464-84a7-47b1-a125-12c65435abeb service nova] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:02:56.945656 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd2e464-84a7-47b1-a125-12c65435abeb service nova] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:02:56.949704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c54f6c-1dd8-4749-8390-fa076a88b79a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:02:56.950313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c54f6c-1dd8-4749-8390-fa076a88b79a service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:02:56.950591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c54f6c-1dd8-4749-8390-fa076a88b79a service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:02:56.950898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c54f6c-1dd8-4749-8390-fa076a88b79a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:02:56.952668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-e6c54f6c-1dd8-4749-8390-fa076a88b79a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:02:57.019500 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c54f6c-1dd8-4749-8390-fa076a88b79a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:02:57.020960 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c54f6c-1dd8-4749-8390-fa076a88b79a service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:02:57.031761 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1077/2113] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:02:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:01.125075 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f272ff-bbe3-4af3-a57d-8044da225107 service nova] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:01.127107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f2f272ff-bbe3-4af3-a57d-8044da225107 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:01.215252 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f272ff-bbe3-4af3-a57d-8044da225107 service nova] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:01.226293 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b777624-442f-4a70-9762-199edd8635da service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87785ccd37a642e08a0ad34a4fb1c754', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:01.228075 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b777624-442f-4a70-9762-199edd8635da service placement] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:01.228588 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b777624-442f-4a70-9762-199edd8635da service placement] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:01.229102 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b777624-442f-4a70-9762-199edd8635da service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:01.233592 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7b777624-442f-4a70-9762-199edd8635da service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:01.250504 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c54f6c-1dd8-4749-8390-fa076a88b79a service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:01.251044 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b777624-442f-4a70-9762-199edd8635da service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:01.252127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-e6c54f6c-1dd8-4749-8390-fa076a88b79a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:01.252250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b777624-442f-4a70-9762-199edd8635da service placement] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:01.268419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1037/2114] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:01 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 01:03:01.287068 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c54f6c-1dd8-4749-8390-fa076a88b79a service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:01.295097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfc65c6-95fc-4033-abcc-be9fed9e283f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:01.296104 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfc65c6-95fc-4033-abcc-be9fed9e283f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:01.296760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfc65c6-95fc-4033-abcc-be9fed9e283f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:01.297369 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfc65c6-95fc-4033-abcc-be9fed9e283f admin admin] PATH_INFO: `/v3/users/b01ce2d27bdb4c91a3f55c891f949e00` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:01.299051 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfc65c6-95fc-4033-abcc-be9fed9e283f admin admin] RBAC: Authorizing `identity:delete_user(user_id=b01ce2d27bdb4c91a3f55c891f949e00)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:01.302690 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfc65c6-95fc-4033-abcc-be9fed9e283f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:01.361372 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1dfc65c6-95fc-4033-abcc-be9fed9e283f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b01ce2d27bdb4c91a3f55c891f949e00'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:01.362047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1dfc65c6-95fc-4033-abcc-be9fed9e283f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b01ce2d27bdb4c91a3f55c891f949e00'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:01.369021 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1dfc65c6-95fc-4033-abcc-be9fed9e283f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b01ce2d27bdb4c91a3f55c891f949e00'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:01.381532 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1dfc65c6-95fc-4033-abcc-be9fed9e283f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b01ce2d27bdb4c91a3f55c891f949e00'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:01.389612 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1dfc65c6-95fc-4033-abcc-be9fed9e283f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b01ce2d27bdb4c91a3f55c891f949e00'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:01.395805 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1078/2115] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:03:01 2022] DELETE /identity/v3/users/b01ce2d27bdb4c91a3f55c891f949e00 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:01.573493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b777624-442f-4a70-9762-199edd8635da service placement] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:01.584521 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b777624-442f-4a70-9762-199edd8635da service placement] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:01.642906 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5edbd0-c050-4317-aaa2-040fa570f217 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:01.643545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5edbd0-c050-4317-aaa2-040fa570f217 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:01.643827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5edbd0-c050-4317-aaa2-040fa570f217 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:01.644104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5edbd0-c050-4317-aaa2-040fa570f217 admin admin] PATH_INFO: `/v3/projects/5462fcbfca114c8885f77f42b479b13d` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:01.645218 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5edbd0-c050-4317-aaa2-040fa570f217 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5462fcbfca114c8885f77f42b479b13d)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:01.647194 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5edbd0-c050-4317-aaa2-040fa570f217 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:01.694930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5e5edbd0-c050-4317-aaa2-040fa570f217 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5462fcbfca114c8885f77f42b479b13d', 'request_id': 'req-5e5edbd0-c050-4317-aaa2-040fa570f217'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:01.704601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5e5edbd0-c050-4317-aaa2-040fa570f217 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5462fcbfca114c8885f77f42b479b13d', 'request_id': 'req-5e5edbd0-c050-4317-aaa2-040fa570f217'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:01.713487 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5e5edbd0-c050-4317-aaa2-040fa570f217 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5462fcbfca114c8885f77f42b479b13d', 'request_id': 'req-5e5edbd0-c050-4317-aaa2-040fa570f217'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:01.733168 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5e5edbd0-c050-4317-aaa2-040fa570f217 admin admin] The token cache is being invalidate because project 5462fcbfca114c8885f77f42b479b13d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:03:01.734267 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5e5edbd0-c050-4317-aaa2-040fa570f217 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:01.735004 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5e5edbd0-c050-4317-aaa2-040fa570f217 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:01.736600 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1038/2116] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:03:01 2022] DELETE /identity/v3/projects/5462fcbfca114c8885f77f42b479b13d => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:01.743832 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba13933f-6244-4b81-a3d5-5cc62c2d146a None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:01.744130 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba13933f-6244-4b81-a3d5-5cc62c2d146a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:01.744408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba13933f-6244-4b81-a3d5-5cc62c2d146a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:01.768351 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-ba13933f-6244-4b81-a3d5-5cc62c2d146a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:01.769547 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-ba13933f-6244-4b81-a3d5-5cc62c2d146a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:01.855829 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:01.871663 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1079/2117] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:01.880266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b6e984-6bb0-44ac-8680-bc4d6b21f5a3 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:01.880553 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b6e984-6bb0-44ac-8680-bc4d6b21f5a3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:01.880853 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b6e984-6bb0-44ac-8680-bc4d6b21f5a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:01.898193 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:01.906572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e5b6e984-6bb0-44ac-8680-bc4d6b21f5a3 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:01.908738 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e5b6e984-6bb0-44ac-8680-bc4d6b21f5a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:01.918575 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1039/2118] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:01.922750 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba13933f-6244-4b81-a3d5-5cc62c2d146a None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:01.933850 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba13933f-6244-4b81-a3d5-5cc62c2d146a None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:01.938614 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e46235-8cc9-4d4a-b995-56313b2409bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:01.939292 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e46235-8cc9-4d4a-b995-56313b2409bb admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:01.939575 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e46235-8cc9-4d4a-b995-56313b2409bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:01.939854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e46235-8cc9-4d4a-b995-56313b2409bb admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:01.940523 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e46235-8cc9-4d4a-b995-56313b2409bb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:03:01.940831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e46235-8cc9-4d4a-b995-56313b2409bb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:01.942491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e46235-8cc9-4d4a-b995-56313b2409bb admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:01.955563 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1080/2119] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:03:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 01:03:01.960373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b6e984-6bb0-44ac-8680-bc4d6b21f5a3 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:01.969802 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b6e984-6bb0-44ac-8680-bc4d6b21f5a3 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:01.974534 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c55a98-e4bd-415b-b081-fe292ebb4ccb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:01.975167 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c55a98-e4bd-415b-b081-fe292ebb4ccb admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:01.975450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c55a98-e4bd-415b-b081-fe292ebb4ccb admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:01.975726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c55a98-e4bd-415b-b081-fe292ebb4ccb admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:01.976355 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c55a98-e4bd-415b-b081-fe292ebb4ccb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:01.979153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c55a98-e4bd-415b-b081-fe292ebb4ccb admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:02.008269 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1040/2120] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:03:01 2022] POST /identity/v3/projects => generated 375 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:03:02.012707 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e46235-8cc9-4d4a-b995-56313b2409bb admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:02.023522 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e46235-8cc9-4d4a-b995-56313b2409bb admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:02.081913 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcdcd8c-274a-4b46-910e-e1c57404005a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:02.082526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcdcd8c-274a-4b46-910e-e1c57404005a admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:02.082805 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcdcd8c-274a-4b46-910e-e1c57404005a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:02.083106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcdcd8c-274a-4b46-910e-e1c57404005a admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:02.083808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcdcd8c-274a-4b46-910e-e1c57404005a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:02.085731 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcdcd8c-274a-4b46-910e-e1c57404005a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:02.107157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1081/2121] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:03:02 2022] POST /identity/v3/users => generated 288 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:03:02.111587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c55a98-e4bd-415b-b081-fe292ebb4ccb admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:02.123264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c55a98-e4bd-415b-b081-fe292ebb4ccb admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:02.128018 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34964a3e-f199-4ec7-a1e1-7d548a9168f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:02.128630 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34964a3e-f199-4ec7-a1e1-7d548a9168f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:02.128916 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34964a3e-f199-4ec7-a1e1-7d548a9168f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:02.129190 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34964a3e-f199-4ec7-a1e1-7d548a9168f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:02.129869 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34964a3e-f199-4ec7-a1e1-7d548a9168f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:02.131518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34964a3e-f199-4ec7-a1e1-7d548a9168f6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:02.141586 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1041/2122] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:03:02 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:03:02.145904 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcdcd8c-274a-4b46-910e-e1c57404005a admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:02.156442 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcdcd8c-274a-4b46-910e-e1c57404005a admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:02.161148 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ed5270-c2af-43f5-89b7-b0479e2d31cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:02.161747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ed5270-c2af-43f5-89b7-b0479e2d31cc admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:02.162040 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ed5270-c2af-43f5-89b7-b0479e2d31cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:02.162316 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ed5270-c2af-43f5-89b7-b0479e2d31cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:02.162987 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ed5270-c2af-43f5-89b7-b0479e2d31cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:02.165099 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ed5270-c2af-43f5-89b7-b0479e2d31cc admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:02.175746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1082/2123] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:03:02 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:03:02.180014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34964a3e-f199-4ec7-a1e1-7d548a9168f6 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:02.190925 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34964a3e-f199-4ec7-a1e1-7d548a9168f6 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:02.195799 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13caedf-8b8d-4add-9439-7ae2d696ee33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:02.196461 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13caedf-8b8d-4add-9439-7ae2d696ee33 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:02.196740 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13caedf-8b8d-4add-9439-7ae2d696ee33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:02.197014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13caedf-8b8d-4add-9439-7ae2d696ee33 admin admin] PATH_INFO: `/v3/projects/b8852bfe31aa4ba68190661d26c177cb/users/e1283100ac2c439cb308b9c57ae31f40/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:02.212422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13caedf-8b8d-4add-9439-7ae2d696ee33 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8852bfe31aa4ba68190661d26c177cb, user_id=e1283100ac2c439cb308b9c57ae31f40, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:02.215139 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13caedf-8b8d-4add-9439-7ae2d696ee33 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:02.228660 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1042/2124] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:03:02 2022] PUT /identity/v3/projects/b8852bfe31aa4ba68190661d26c177cb/users/e1283100ac2c439cb308b9c57ae31f40/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:02.233914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c45083-736b-4eb7-8b11-7c1d4445d9ce None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:02.234189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c45083-736b-4eb7-8b11-7c1d4445d9ce None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:02.234475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c45083-736b-4eb7-8b11-7c1d4445d9ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:02.254213 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-a1c45083-736b-4eb7-8b11-7c1d4445d9ce None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:02.255455 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-a1c45083-736b-4eb7-8b11-7c1d4445d9ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:02.324352 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1083/2125] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:02.334108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13caedf-8b8d-4add-9439-7ae2d696ee33 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:02.335856 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e13caedf-8b8d-4add-9439-7ae2d696ee33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:02.401649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13caedf-8b8d-4add-9439-7ae2d696ee33 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:02.407444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e247abd9-0595-459f-bf9e-d2e78f8047d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:02.408150 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e247abd9-0595-459f-bf9e-d2e78f8047d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:02.408433 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e247abd9-0595-459f-bf9e-d2e78f8047d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:02.408737 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e247abd9-0595-459f-bf9e-d2e78f8047d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:02.420143 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e247abd9-0595-459f-bf9e-d2e78f8047d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:02.421545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e247abd9-0595-459f-bf9e-d2e78f8047d4 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:02.439842 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1043/2126] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 01:03:06.812222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d390d55-e1eb-42f0-a965-2d284181e031 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:06.812727 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d390d55-e1eb-42f0-a965-2d284181e031 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:06.812845 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d390d55-e1eb-42f0-a965-2d284181e031 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:06.855688 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-2d390d55-e1eb-42f0-a965-2d284181e031 None None] MFA Rules not processed for user `e1283100ac2c439cb308b9c57ae31f40`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:06.856999 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-2d390d55-e1eb-42f0-a965-2d284181e031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:06.916799 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:06.937667 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1084/2127] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:06 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 01:03:06.953467 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e247abd9-0595-459f-bf9e-d2e78f8047d4 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:06.955266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e247abd9-0595-459f-bf9e-d2e78f8047d4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:07.066632 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e247abd9-0595-459f-bf9e-d2e78f8047d4 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:07.078809 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f25b74-3cf7-4e7e-a461-114730b4855a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:07.079584 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f25b74-3cf7-4e7e-a461-114730b4855a service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:07.079935 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f25b74-3cf7-4e7e-a461-114730b4855a service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:07.080290 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f25b74-3cf7-4e7e-a461-114730b4855a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:07.089259 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f25b74-3cf7-4e7e-a461-114730b4855a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:07.090929 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f25b74-3cf7-4e7e-a461-114730b4855a service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:07.101262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1044/2128] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:06 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:09.049681 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd023d4f-55ba-4904-9f3f-fab647e5a6f0 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:09.050192 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd023d4f-55ba-4904-9f3f-fab647e5a6f0 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:09.050276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd023d4f-55ba-4904-9f3f-fab647e5a6f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:09.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-bd023d4f-55ba-4904-9f3f-fab647e5a6f0 None None] MFA Rules not processed for user `e1283100ac2c439cb308b9c57ae31f40`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:09.120602 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-bd023d4f-55ba-4904-9f3f-fab647e5a6f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:09.142572 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1085/2129] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:09 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 01:03:09.630308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f25b74-3cf7-4e7e-a461-114730b4855a service nova] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:09.659461 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f25b74-3cf7-4e7e-a461-114730b4855a service nova] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:09.665228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f5eba5-eba1-4d8e-832e-230060707904 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:09.666215 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f5eba5-eba1-4d8e-832e-230060707904 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:09.666690 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f5eba5-eba1-4d8e-832e-230060707904 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:09.667210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f5eba5-eba1-4d8e-832e-230060707904 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:09.669542 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c9f5eba5-eba1-4d8e-832e-230060707904 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:09.681087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f5eba5-eba1-4d8e-832e-230060707904 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:09.682856 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f5eba5-eba1-4d8e-832e-230060707904 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:09.695284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1045/2130] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:09 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:10.169147 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd023d4f-55ba-4904-9f3f-fab647e5a6f0 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:10.170754 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-bd023d4f-55ba-4904-9f3f-fab647e5a6f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:10.334115 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd023d4f-55ba-4904-9f3f-fab647e5a6f0 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:10.342707 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf91277-a60d-4c48-8c24-9503554fbd06 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87785ccd37a642e08a0ad34a4fb1c754', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:10.343901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf91277-a60d-4c48-8c24-9503554fbd06 service placement] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:10.344265 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf91277-a60d-4c48-8c24-9503554fbd06 service placement] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:10.344788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf91277-a60d-4c48-8c24-9503554fbd06 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:10.348950 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-fcf91277-a60d-4c48-8c24-9503554fbd06 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:10.365506 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf91277-a60d-4c48-8c24-9503554fbd06 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:10.368410 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf91277-a60d-4c48-8c24-9503554fbd06 service placement] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:10.382024 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1086/2131] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:10 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:11.529769 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca46bd66-95f5-43a3-ab60-7be54f314080 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:11.531942 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca46bd66-95f5-43a3-ab60-7be54f314080 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:11.532765 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca46bd66-95f5-43a3-ab60-7be54f314080 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:11.580253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ca46bd66-95f5-43a3-ab60-7be54f314080 None None] MFA Rules not processed for user `c4e65cef89af48a7a51e5548634de75f`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:11.582710 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ca46bd66-95f5-43a3-ab60-7be54f314080 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:11.611083 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:11.687568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1046/2132] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:11 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:12.172549 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf91277-a60d-4c48-8c24-9503554fbd06 service placement] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:12.175125 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-fcf91277-a60d-4c48-8c24-9503554fbd06 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:12.191128 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf91277-a60d-4c48-8c24-9503554fbd06 service placement] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:12.196544 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e09d0e-1456-4cfa-92fc-ce21f5f3543c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:12.197196 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e09d0e-1456-4cfa-92fc-ce21f5f3543c service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:12.197477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e09d0e-1456-4cfa-92fc-ce21f5f3543c service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:12.197994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e09d0e-1456-4cfa-92fc-ce21f5f3543c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:12.200370 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b1e09d0e-1456-4cfa-92fc-ce21f5f3543c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:12.212489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e09d0e-1456-4cfa-92fc-ce21f5f3543c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:12.214184 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e09d0e-1456-4cfa-92fc-ce21f5f3543c service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:12.224878 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1087/2133] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:13.286448 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca46bd66-95f5-43a3-ab60-7be54f314080 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:13.325427 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca46bd66-95f5-43a3-ab60-7be54f314080 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:13.330648 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5973240-6ca5-498a-ac68-496fd44ba6a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:13.331587 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5973240-6ca5-498a-ac68-496fd44ba6a9 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:13.331890 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5973240-6ca5-498a-ac68-496fd44ba6a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:13.332230 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5973240-6ca5-498a-ac68-496fd44ba6a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:13.334279 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f5973240-6ca5-498a-ac68-496fd44ba6a9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:13.351201 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5973240-6ca5-498a-ac68-496fd44ba6a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:13.353617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5973240-6ca5-498a-ac68-496fd44ba6a9 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:13.380701 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1047/2134] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:13 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:13.551257 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4889aba6-fca5-4d8a-b9f0-8cdb5db8fc6f None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:13.552176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4889aba6-fca5-4d8a-b9f0-8cdb5db8fc6f None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:13.552176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4889aba6-fca5-4d8a-b9f0-8cdb5db8fc6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:13.591470 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-4889aba6-fca5-4d8a-b9f0-8cdb5db8fc6f None None] MFA Rules not processed for user `33b136afbf8a4d98a9d62053b68707e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:13.592723 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4889aba6-fca5-4d8a-b9f0-8cdb5db8fc6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:13.718384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1088/2135] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 16 01:03:13 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:13.769420 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5973240-6ca5-498a-ac68-496fd44ba6a9 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:13.771645 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f5973240-6ca5-498a-ac68-496fd44ba6a9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:13.866028 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5973240-6ca5-498a-ac68-496fd44ba6a9 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:13.873646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478f463f-d6f9-485c-be05-19d406e64428 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de64d498da78448394b0d4d1851a2789', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:13.874402 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478f463f-d6f9-485c-be05-19d406e64428 service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:13.874750 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478f463f-d6f9-485c-be05-19d406e64428 service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:13.875067 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478f463f-d6f9-485c-be05-19d406e64428 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:13.885394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478f463f-d6f9-485c-be05-19d406e64428 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:13.887339 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478f463f-d6f9-485c-be05-19d406e64428 service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:13.904855 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1048/2136] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 16 01:03:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 16 01:03:14.611355 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4889aba6-fca5-4d8a-b9f0-8cdb5db8fc6f None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:14.639724 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4889aba6-fca5-4d8a-b9f0-8cdb5db8fc6f None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:14.654852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c87630-cc4b-4492-83e7-2d6f6479ca4b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:14.656698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c87630-cc4b-4492-83e7-2d6f6479ca4b service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:14.657624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c87630-cc4b-4492-83e7-2d6f6479ca4b service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:14.666538 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c87630-cc4b-4492-83e7-2d6f6479ca4b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:14.680881 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-e0c87630-cc4b-4492-83e7-2d6f6479ca4b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:14.720004 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c87630-cc4b-4492-83e7-2d6f6479ca4b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:14.722007 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c87630-cc4b-4492-83e7-2d6f6479ca4b service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:14.762146 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1089/2137] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:23.893395 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=55 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:24.752791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=77 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:32.196563 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbb4203-1be9-4796-b9a6-bc4f09be237d None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:32.197068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbb4203-1be9-4796-b9a6-bc4f09be237d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:32.197155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbb4203-1be9-4796-b9a6-bc4f09be237d None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:32.198712 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1049/2138] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 16 01:03:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 01:03:32.207765 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71ac43b-875f-4c5c-8072-fd3ae8e8a9e3 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:32.209632 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71ac43b-875f-4c5c-8072-fd3ae8e8a9e3 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:32.210293 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71ac43b-875f-4c5c-8072-fd3ae8e8a9e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:32.236403 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-c71ac43b-875f-4c5c-8072-fd3ae8e8a9e3 None None] MFA Rules not processed for user `4eba26813e154595a12a67fe4be5eeff`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:32.237655 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c71ac43b-875f-4c5c-8072-fd3ae8e8a9e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:32.248655 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1090/2139] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 16 01:03:32 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:32.262121 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbb4203-1be9-4796-b9a6-bc4f09be237d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:32.264266 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fdbb4203-1be9-4796-b9a6-bc4f09be237d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:32.339333 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbb4203-1be9-4796-b9a6-bc4f09be237d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:32.343397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4c4486-c826-44dc-8ecc-2bceddf6e409 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '78473c29cef449b89bc9185de67f1bf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:32.344791 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c4486-c826-44dc-8ecc-2bceddf6e409 service glance] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:32.345240 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c4486-c826-44dc-8ecc-2bceddf6e409 service glance] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:32.345756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c4486-c826-44dc-8ecc-2bceddf6e409 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:32.358892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4c4486-c826-44dc-8ecc-2bceddf6e409 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:32.360430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4c4486-c826-44dc-8ecc-2bceddf6e409 service glance] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:32.372714 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1050/2140] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:32.447558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5268104f-41b2-423b-aa3c-a2fa064adc7a None None] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:32.448282 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5268104f-41b2-423b-aa3c-a2fa064adc7a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:32.448754 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5268104f-41b2-423b-aa3c-a2fa064adc7a None None] PATH_INFO: `/` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:32.450120 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1091/2141] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 16 01:03:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 01:03:32.455730 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5118c59-0705-43d8-acbf-255a9ab7d71e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:32.456016 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5118c59-0705-43d8-acbf-255a9ab7d71e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:32.456304 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5118c59-0705-43d8-acbf-255a9ab7d71e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:32.478954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e5118c59-0705-43d8-acbf-255a9ab7d71e None None] MFA Rules not processed for user `f4eb1a640186411097cb1e3bb9e92020`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:32.480401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e5118c59-0705-43d8-acbf-255a9ab7d71e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:32.487943 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1051/2142] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 16 01:03:32 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:32.500305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5268104f-41b2-423b-aa3c-a2fa064adc7a None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:32.511760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5268104f-41b2-423b-aa3c-a2fa064adc7a None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:32.519607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf6a525-ccf7-441c-9aa6-ce65581c0f63 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:32.520365 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf6a525-ccf7-441c-9aa6-ce65581c0f63 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:32.521002 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf6a525-ccf7-441c-9aa6-ce65581c0f63 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:32.521483 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf6a525-ccf7-441c-9aa6-ce65581c0f63 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:32.531241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf6a525-ccf7-441c-9aa6-ce65581c0f63 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:32.532692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf6a525-ccf7-441c-9aa6-ce65581c0f63 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:32.541799 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1092/2143] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:32 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:32.626879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca947bf8-e0ae-43ca-8427-e1b53f5c0bbf None None] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:32.627183 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca947bf8-e0ae-43ca-8427-e1b53f5c0bbf None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:32.627460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca947bf8-e0ae-43ca-8427-e1b53f5c0bbf None None] PATH_INFO: `/` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:32.629287 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1052/2144] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 16 01:03:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 16 01:03:32.636111 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d873d4a-d4a7-4bb9-91a2-4e6e15e8c665 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:32.636846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d873d4a-d4a7-4bb9-91a2-4e6e15e8c665 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:32.637193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d873d4a-d4a7-4bb9-91a2-4e6e15e8c665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:32.659479 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-8d873d4a-d4a7-4bb9-91a2-4e6e15e8c665 None None] MFA Rules not processed for user `4eba26813e154595a12a67fe4be5eeff`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:32.660736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8d873d4a-d4a7-4bb9-91a2-4e6e15e8c665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:32.670825 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1093/2145] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 16 01:03:32 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:32.682152 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca947bf8-e0ae-43ca-8427-e1b53f5c0bbf None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:32.693172 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca947bf8-e0ae-43ca-8427-e1b53f5c0bbf None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:32.698308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d9b88f-56e7-46da-8653-2d16f93a0718 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:32.699054 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d9b88f-56e7-46da-8653-2d16f93a0718 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:32.699390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d9b88f-56e7-46da-8653-2d16f93a0718 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:32.699724 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d9b88f-56e7-46da-8653-2d16f93a0718 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:32.710460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d9b88f-56e7-46da-8653-2d16f93a0718 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:32.712025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d9b88f-56e7-46da-8653-2d16f93a0718 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:32.723568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1053/2146] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:34.209525 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d873d4a-d4a7-4bb9-91a2-4e6e15e8c665 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:34.212063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8d873d4a-d4a7-4bb9-91a2-4e6e15e8c665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:34.234296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d873d4a-d4a7-4bb9-91a2-4e6e15e8c665 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:34.240772 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8819a626-8771-4c4f-8d20-0ea6d45bbbe1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:34.241672 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8819a626-8771-4c4f-8d20-0ea6d45bbbe1 service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:34.242165 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8819a626-8771-4c4f-8d20-0ea6d45bbbe1 service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:34.242541 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8819a626-8771-4c4f-8d20-0ea6d45bbbe1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:34.244539 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8819a626-8771-4c4f-8d20-0ea6d45bbbe1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:34.259941 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8819a626-8771-4c4f-8d20-0ea6d45bbbe1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:34.262322 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8819a626-8771-4c4f-8d20-0ea6d45bbbe1 service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:34.275877 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1094/2147] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:37.365840 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03c4045-d9ad-405c-b628-1132fc8c84d8 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:37.366909 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03c4045-d9ad-405c-b628-1132fc8c84d8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:37.367079 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03c4045-d9ad-405c-b628-1132fc8c84d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:37.387050 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-e03c4045-d9ad-405c-b628-1132fc8c84d8 None None] MFA Rules not processed for user `33b136afbf8a4d98a9d62053b68707e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:37.388395 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e03c4045-d9ad-405c-b628-1132fc8c84d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:37.395889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1054/2148] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 16 01:03:37 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:37.409219 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8819a626-8771-4c4f-8d20-0ea6d45bbbe1 service nova] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:37.422746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8819a626-8771-4c4f-8d20-0ea6d45bbbe1 service nova] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:37.436695 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34dfd13-ad68-45de-9c3f-2a6475637420 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de64d498da78448394b0d4d1851a2789', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:37.437338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34dfd13-ad68-45de-9c3f-2a6475637420 service swift] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:37.437690 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34dfd13-ad68-45de-9c3f-2a6475637420 service swift] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:37.438188 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34dfd13-ad68-45de-9c3f-2a6475637420 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:37.452612 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34dfd13-ad68-45de-9c3f-2a6475637420 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:37.454563 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34dfd13-ad68-45de-9c3f-2a6475637420 service swift] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:37.468827 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1095/2149] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 16 01:03:37 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 16 01:03:42.031121 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03c4045-d9ad-405c-b628-1132fc8c84d8 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:42.041337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03c4045-d9ad-405c-b628-1132fc8c84d8 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:42.045238 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fda237f-500c-40df-8b2c-16962c4c65ee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:42.047238 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fda237f-500c-40df-8b2c-16962c4c65ee service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:42.047749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fda237f-500c-40df-8b2c-16962c4c65ee service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:42.048526 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fda237f-500c-40df-8b2c-16962c4c65ee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:42.051226 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9fda237f-500c-40df-8b2c-16962c4c65ee service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:42.078391 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fda237f-500c-40df-8b2c-16962c4c65ee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:42.079952 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fda237f-500c-40df-8b2c-16962c4c65ee service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:42.098472 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1055/2150] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:47.461464 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=24 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:47.699139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34dfd13-ad68-45de-9c3f-2a6475637420 service swift] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:47.710555 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34dfd13-ad68-45de-9c3f-2a6475637420 service swift] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:47.715968 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a03e30-ba62-4a8a-91b2-7f2942281553 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87785ccd37a642e08a0ad34a4fb1c754', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:47.716606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a03e30-ba62-4a8a-91b2-7f2942281553 service placement] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:47.716892 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a03e30-ba62-4a8a-91b2-7f2942281553 service placement] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:47.717165 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a03e30-ba62-4a8a-91b2-7f2942281553 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:47.719281 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-88a03e30-ba62-4a8a-91b2-7f2942281553 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:47.731899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a03e30-ba62-4a8a-91b2-7f2942281553 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:47.733504 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a03e30-ba62-4a8a-91b2-7f2942281553 service placement] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:47.744593 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1096/2151] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:47 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:47.763947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fda237f-500c-40df-8b2c-16962c4c65ee service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:47.765495 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9fda237f-500c-40df-8b2c-16962c4c65ee service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:47.780646 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fda237f-500c-40df-8b2c-16962c4c65ee service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:47.786889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501d8184-69ea-444c-8c21-897342ed7add admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:47.788038 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501d8184-69ea-444c-8c21-897342ed7add admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:47.788323 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501d8184-69ea-444c-8c21-897342ed7add admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:47.788696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501d8184-69ea-444c-8c21-897342ed7add admin admin] PATH_INFO: `/v3/users/466f2f8ddd184a98bdb5e922818905cc` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:47.789848 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501d8184-69ea-444c-8c21-897342ed7add admin admin] RBAC: Authorizing `identity:delete_user(user_id=466f2f8ddd184a98bdb5e922818905cc)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:47.791950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501d8184-69ea-444c-8c21-897342ed7add admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:47.840399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-501d8184-69ea-444c-8c21-897342ed7add admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '466f2f8ddd184a98bdb5e922818905cc'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:47.841076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-501d8184-69ea-444c-8c21-897342ed7add admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '466f2f8ddd184a98bdb5e922818905cc'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:47.848427 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-501d8184-69ea-444c-8c21-897342ed7add admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '466f2f8ddd184a98bdb5e922818905cc'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:47.849057 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-501d8184-69ea-444c-8c21-897342ed7add admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '466f2f8ddd184a98bdb5e922818905cc'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:47.862471 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-501d8184-69ea-444c-8c21-897342ed7add admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '466f2f8ddd184a98bdb5e922818905cc'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:47.875559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1056/2152] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:03:47 2022] DELETE /identity/v3/users/466f2f8ddd184a98bdb5e922818905cc => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:48.059480 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a03e30-ba62-4a8a-91b2-7f2942281553 service placement] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:48.076498 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a03e30-ba62-4a8a-91b2-7f2942281553 service placement] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:48.135159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09d193e-a08d-43a6-b33a-7f5d49099687 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:48.135799 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09d193e-a08d-43a6-b33a-7f5d49099687 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:48.136079 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09d193e-a08d-43a6-b33a-7f5d49099687 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:48.136353 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09d193e-a08d-43a6-b33a-7f5d49099687 admin admin] PATH_INFO: `/v3/projects/fb3c758e27684dcd9265e2d60036d80b` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:48.138078 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09d193e-a08d-43a6-b33a-7f5d49099687 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb3c758e27684dcd9265e2d60036d80b)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:48.139476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09d193e-a08d-43a6-b33a-7f5d49099687 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:48.187315 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e09d193e-a08d-43a6-b33a-7f5d49099687 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb3c758e27684dcd9265e2d60036d80b', 'request_id': 'req-e09d193e-a08d-43a6-b33a-7f5d49099687'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:48.196040 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e09d193e-a08d-43a6-b33a-7f5d49099687 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb3c758e27684dcd9265e2d60036d80b', 'request_id': 'req-e09d193e-a08d-43a6-b33a-7f5d49099687'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:48.202148 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e09d193e-a08d-43a6-b33a-7f5d49099687 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb3c758e27684dcd9265e2d60036d80b', 'request_id': 'req-e09d193e-a08d-43a6-b33a-7f5d49099687'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:48.215585 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e09d193e-a08d-43a6-b33a-7f5d49099687 admin admin] The token cache is being invalidate because project fb3c758e27684dcd9265e2d60036d80b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:03:48.215884 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e09d193e-a08d-43a6-b33a-7f5d49099687 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:48.216567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e09d193e-a08d-43a6-b33a-7f5d49099687 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:48.218079 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1097/2153] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:03:48 2022] DELETE /identity/v3/projects/fb3c758e27684dcd9265e2d60036d80b => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:48.223036 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9271c161-30d3-4b35-bd30-7325522cc051 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:48.223338 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9271c161-30d3-4b35-bd30-7325522cc051 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:48.223607 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9271c161-30d3-4b35-bd30-7325522cc051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:48.255711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-9271c161-30d3-4b35-bd30-7325522cc051 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:48.257304 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9271c161-30d3-4b35-bd30-7325522cc051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:48.344064 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:48.407937 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1057/2154] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:48.408187 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be94f479-0b76-4840-bc7c-c2053bca9142 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:48.408354 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be94f479-0b76-4840-bc7c-c2053bca9142 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:48.408635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be94f479-0b76-4840-bc7c-c2053bca9142 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:48.444296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-be94f479-0b76-4840-bc7c-c2053bca9142 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:48.446736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-be94f479-0b76-4840-bc7c-c2053bca9142 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:48.468331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1098/2155] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 01:03:48.472448 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9271c161-30d3-4b35-bd30-7325522cc051 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:48.484092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9271c161-30d3-4b35-bd30-7325522cc051 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:48.489331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f94d9e-e27f-4055-a52f-2a4d4d54ffb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:48.490098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f94d9e-e27f-4055-a52f-2a4d4d54ffb0 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:48.490801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f94d9e-e27f-4055-a52f-2a4d4d54ffb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:48.491223 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f94d9e-e27f-4055-a52f-2a4d4d54ffb0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:48.491988 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f94d9e-e27f-4055-a52f-2a4d4d54ffb0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:03:48.492767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f94d9e-e27f-4055-a52f-2a4d4d54ffb0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:48.494845 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f94d9e-e27f-4055-a52f-2a4d4d54ffb0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:48.508886 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1058/2156] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:03:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:03:48.513057 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be94f479-0b76-4840-bc7c-c2053bca9142 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:48.523626 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be94f479-0b76-4840-bc7c-c2053bca9142 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:48.528539 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91b387a-7ca7-4733-b020-6649f9250cad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:48.529278 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91b387a-7ca7-4733-b020-6649f9250cad admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:48.529657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91b387a-7ca7-4733-b020-6649f9250cad admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:48.530031 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91b387a-7ca7-4733-b020-6649f9250cad admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:48.530767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91b387a-7ca7-4733-b020-6649f9250cad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:48.533087 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91b387a-7ca7-4733-b020-6649f9250cad admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:48.567869 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1099/2157] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:03:48 2022] POST /identity/v3/projects => generated 365 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:03:48.569805 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f94d9e-e27f-4055-a52f-2a4d4d54ffb0 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:48.581395 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f94d9e-e27f-4055-a52f-2a4d4d54ffb0 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:48.654354 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c922af2-3b0d-4b49-9561-f3dd6a4f53f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:48.655011 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c922af2-3b0d-4b49-9561-f3dd6a4f53f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:48.655414 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c922af2-3b0d-4b49-9561-f3dd6a4f53f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:48.655701 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c922af2-3b0d-4b49-9561-f3dd6a4f53f6 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:48.656343 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c922af2-3b0d-4b49-9561-f3dd6a4f53f6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:48.658499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c922af2-3b0d-4b49-9561-f3dd6a4f53f6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:48.689977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1059/2158] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:03:48 2022] POST /identity/v3/users => generated 283 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:03:48.695543 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91b387a-7ca7-4733-b020-6649f9250cad admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:48.717241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91b387a-7ca7-4733-b020-6649f9250cad admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:48.727445 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acbfb4d-79eb-4f6e-bd33-96d4c8441a26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:48.728458 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acbfb4d-79eb-4f6e-bd33-96d4c8441a26 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:48.728935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acbfb4d-79eb-4f6e-bd33-96d4c8441a26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:48.729605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acbfb4d-79eb-4f6e-bd33-96d4c8441a26 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:48.730605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acbfb4d-79eb-4f6e-bd33-96d4c8441a26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:48.734506 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acbfb4d-79eb-4f6e-bd33-96d4c8441a26 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:48.758988 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1100/2159] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:03:48 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:03:48.761560 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c922af2-3b0d-4b49-9561-f3dd6a4f53f6 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:48.776771 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c922af2-3b0d-4b49-9561-f3dd6a4f53f6 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:48.781549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ae69cd-8e85-4297-8edd-414d57d19490 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:48.782161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ae69cd-8e85-4297-8edd-414d57d19490 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:48.782635 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ae69cd-8e85-4297-8edd-414d57d19490 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:48.782796 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ae69cd-8e85-4297-8edd-414d57d19490 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:48.783450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ae69cd-8e85-4297-8edd-414d57d19490 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:48.785094 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ae69cd-8e85-4297-8edd-414d57d19490 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:48.796897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1060/2160] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:03:48 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:03:48.801178 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acbfb4d-79eb-4f6e-bd33-96d4c8441a26 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:48.812670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acbfb4d-79eb-4f6e-bd33-96d4c8441a26 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:48.817652 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6ae092-c35c-45ac-b10d-cf6f503159c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:48.818326 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6ae092-c35c-45ac-b10d-cf6f503159c5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:48.819879 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6ae092-c35c-45ac-b10d-cf6f503159c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:48.820166 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6ae092-c35c-45ac-b10d-cf6f503159c5 admin admin] PATH_INFO: `/v3/projects/30ceb8cab7454065b963967e35dde6e2/users/decb891f5bd847be854b9dbedc13d750/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:48.838666 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6ae092-c35c-45ac-b10d-cf6f503159c5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=30ceb8cab7454065b963967e35dde6e2, user_id=decb891f5bd847be854b9dbedc13d750, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:48.842614 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6ae092-c35c-45ac-b10d-cf6f503159c5 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:48.857125 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1101/2161] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:03:48 2022] PUT /identity/v3/projects/30ceb8cab7454065b963967e35dde6e2/users/decb891f5bd847be854b9dbedc13d750/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:48.868277 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e37d501-1716-4e18-8287-b0d5e8d7d3db None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:48.868379 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e37d501-1716-4e18-8287-b0d5e8d7d3db None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:48.868647 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e37d501-1716-4e18-8287-b0d5e8d7d3db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:48.905432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-8e37d501-1716-4e18-8287-b0d5e8d7d3db None None] MFA Rules not processed for user `decb891f5bd847be854b9dbedc13d750`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:48.906888 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-8e37d501-1716-4e18-8287-b0d5e8d7d3db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:48.994886 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1061/2162] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:48 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:48.999275 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6ae092-c35c-45ac-b10d-cf6f503159c5 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:49.014921 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:49.092056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6ae092-c35c-45ac-b10d-cf6f503159c5 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:49.098441 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd12179-7720-4424-aac8-c6a7f0435f03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:49.099255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd12179-7720-4424-aac8-c6a7f0435f03 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:49.107263 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd12179-7720-4424-aac8-c6a7f0435f03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:49.107548 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd12179-7720-4424-aac8-c6a7f0435f03 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:49.108248 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd12179-7720-4424-aac8-c6a7f0435f03 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:49.110207 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd12179-7720-4424-aac8-c6a7f0435f03 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:49.151728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1102/2163] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:03:48 2022] POST /identity/v3/projects => generated 365 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:03:49.153001 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e37d501-1716-4e18-8287-b0d5e8d7d3db None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:49.166497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e37d501-1716-4e18-8287-b0d5e8d7d3db None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:49.199960 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:49.234090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab94728-fa9b-4a55-a1a7-dd01b87510dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:49.234622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab94728-fa9b-4a55-a1a7-dd01b87510dd admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:49.234932 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab94728-fa9b-4a55-a1a7-dd01b87510dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:49.235212 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab94728-fa9b-4a55-a1a7-dd01b87510dd admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:49.235848 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab94728-fa9b-4a55-a1a7-dd01b87510dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:49.238492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab94728-fa9b-4a55-a1a7-dd01b87510dd admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:49.260975 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1062/2164] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:03:49 2022] POST /identity/v3/users => generated 289 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:03:49.264983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd12179-7720-4424-aac8-c6a7f0435f03 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:49.275958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd12179-7720-4424-aac8-c6a7f0435f03 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:49.281479 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b785e49d-5f4d-4ebb-9939-49fee9574976 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:49.282248 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b785e49d-5f4d-4ebb-9939-49fee9574976 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:49.282628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b785e49d-5f4d-4ebb-9939-49fee9574976 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:49.283024 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b785e49d-5f4d-4ebb-9939-49fee9574976 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:49.283773 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b785e49d-5f4d-4ebb-9939-49fee9574976 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:49.285512 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b785e49d-5f4d-4ebb-9939-49fee9574976 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:49.300767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1103/2165] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:03:49 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:03:49.301614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab94728-fa9b-4a55-a1a7-dd01b87510dd admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:49.312687 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab94728-fa9b-4a55-a1a7-dd01b87510dd admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:49.317378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5159fcd6-da7a-4ade-95b2-d7f039e55365 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:49.318034 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5159fcd6-da7a-4ade-95b2-d7f039e55365 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:49.318612 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5159fcd6-da7a-4ade-95b2-d7f039e55365 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:49.319015 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5159fcd6-da7a-4ade-95b2-d7f039e55365 admin admin] PATH_INFO: `/v3/projects/d65efccc51b242328695e268c0200498/users/f49398aeac73483592cdffa25bfd4613/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:49.335708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5159fcd6-da7a-4ade-95b2-d7f039e55365 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d65efccc51b242328695e268c0200498, user_id=f49398aeac73483592cdffa25bfd4613, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:49.338677 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5159fcd6-da7a-4ade-95b2-d7f039e55365 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:49.352985 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1063/2166] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:03:49 2022] PUT /identity/v3/projects/d65efccc51b242328695e268c0200498/users/f49398aeac73483592cdffa25bfd4613/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:03:49.357711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b785e49d-5f4d-4ebb-9939-49fee9574976 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:49.369558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b785e49d-5f4d-4ebb-9939-49fee9574976 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:49.459702 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1e0324-c48e-486c-b997-734aba207bb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:49.460720 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1e0324-c48e-486c-b997-734aba207bb5 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:49.461094 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1e0324-c48e-486c-b997-734aba207bb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:49.461751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1e0324-c48e-486c-b997-734aba207bb5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:49.462937 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1e0324-c48e-486c-b997-734aba207bb5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:49.465328 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1e0324-c48e-486c-b997-734aba207bb5 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:49.484184 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1104/2167] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:03:49 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:03:49.485370 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5159fcd6-da7a-4ade-95b2-d7f039e55365 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:49.499488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5159fcd6-da7a-4ade-95b2-d7f039e55365 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:49.507872 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c879601-6c6e-4e07-889b-93f0b134a5ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:49.508871 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c879601-6c6e-4e07-889b-93f0b134a5ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:49.509155 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c879601-6c6e-4e07-889b-93f0b134a5ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:49.509435 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c879601-6c6e-4e07-889b-93f0b134a5ea admin admin] PATH_INFO: `/v3/projects/d65efccc51b242328695e268c0200498/users/f49398aeac73483592cdffa25bfd4613/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:49.511719 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c879601-6c6e-4e07-889b-93f0b134a5ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=d65efccc51b242328695e268c0200498, user_id=f49398aeac73483592cdffa25bfd4613, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:49.514353 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c879601-6c6e-4e07-889b-93f0b134a5ea admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:49.538986 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1064/2168] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:03:49 2022] PUT /identity/v3/projects/d65efccc51b242328695e268c0200498/users/f49398aeac73483592cdffa25bfd4613/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:03:49.543369 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f8b9da-9ca1-4427-ac60-3da797de0714 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:49.543710 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f8b9da-9ca1-4427-ac60-3da797de0714 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:49.544019 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f8b9da-9ca1-4427-ac60-3da797de0714 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:49.587211 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-78f8b9da-9ca1-4427-ac60-3da797de0714 None None] MFA Rules not processed for user `f49398aeac73483592cdffa25bfd4613`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:49.588872 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-78f8b9da-9ca1-4427-ac60-3da797de0714 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:49.713553 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1105/2169] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:49 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:49.725170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c879601-6c6e-4e07-889b-93f0b134a5ea admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:49.727068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2c879601-6c6e-4e07-889b-93f0b134a5ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:49.827066 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c879601-6c6e-4e07-889b-93f0b134a5ea admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:49.833163 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f287c8a-21bc-4d84-bfbe-4c625bb0ec6c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:49.833749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f287c8a-21bc-4d84-bfbe-4c625bb0ec6c service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:49.834090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f287c8a-21bc-4d84-bfbe-4c625bb0ec6c service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:49.834907 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f287c8a-21bc-4d84-bfbe-4c625bb0ec6c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:49.846167 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f287c8a-21bc-4d84-bfbe-4c625bb0ec6c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:49.848417 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f287c8a-21bc-4d84-bfbe-4c625bb0ec6c service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:49.861110 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1065/2170] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:49 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:49.944329 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f8b9da-9ca1-4427-ac60-3da797de0714 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:49.954982 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f8b9da-9ca1-4427-ac60-3da797de0714 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:49.958680 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90ce0b8-4d9c-4319-910d-fc474269c100 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:49.959547 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90ce0b8-4d9c-4319-910d-fc474269c100 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:49.959904 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90ce0b8-4d9c-4319-910d-fc474269c100 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:49.960180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90ce0b8-4d9c-4319-910d-fc474269c100 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:49.965601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:50.025722 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90ce0b8-4d9c-4319-910d-fc474269c100 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:50.027402 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90ce0b8-4d9c-4319-910d-fc474269c100 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:50.039109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1106/2171] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:49 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:52.796885 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f287c8a-21bc-4d84-bfbe-4c625bb0ec6c service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:52.863218 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f287c8a-21bc-4d84-bfbe-4c625bb0ec6c service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:52.869282 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c249c73-eb1b-483f-bbc1-11281a2539e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:52.869949 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c249c73-eb1b-483f-bbc1-11281a2539e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:52.870229 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c249c73-eb1b-483f-bbc1-11281a2539e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:52.870501 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c249c73-eb1b-483f-bbc1-11281a2539e0 admin admin] PATH_INFO: `/v3/users/decb891f5bd847be854b9dbedc13d750` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:52.871687 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c249c73-eb1b-483f-bbc1-11281a2539e0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=decb891f5bd847be854b9dbedc13d750)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:52.873630 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c249c73-eb1b-483f-bbc1-11281a2539e0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:52.903542 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:52.924979 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1c249c73-eb1b-483f-bbc1-11281a2539e0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'decb891f5bd847be854b9dbedc13d750'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:52.925578 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1c249c73-eb1b-483f-bbc1-11281a2539e0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'decb891f5bd847be854b9dbedc13d750'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:52.933525 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1c249c73-eb1b-483f-bbc1-11281a2539e0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'decb891f5bd847be854b9dbedc13d750'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:52.934142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1c249c73-eb1b-483f-bbc1-11281a2539e0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'decb891f5bd847be854b9dbedc13d750'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:52.947240 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1c249c73-eb1b-483f-bbc1-11281a2539e0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'decb891f5bd847be854b9dbedc13d750'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:52.959281 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1066/2172] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:03:52 2022] DELETE /identity/v3/users/decb891f5bd847be854b9dbedc13d750 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:52.963831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafe045a-aa59-4c2a-b1ed-fd63d9989d2b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:52.964338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafe045a-aa59-4c2a-b1ed-fd63d9989d2b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:52.964720 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafe045a-aa59-4c2a-b1ed-fd63d9989d2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:52.986076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-bafe045a-aa59-4c2a-b1ed-fd63d9989d2b None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:52.987357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-bafe045a-aa59-4c2a-b1ed-fd63d9989d2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:53.047341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1107/2173] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:53.059480 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c249c73-eb1b-483f-bbc1-11281a2539e0 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:53.060898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-1c249c73-eb1b-483f-bbc1-11281a2539e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:53.119183 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c249c73-eb1b-483f-bbc1-11281a2539e0 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:53.122820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ce5837-4f6b-4ada-9806-36ef8c668df9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:53.123451 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ce5837-4f6b-4ada-9806-36ef8c668df9 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:53.123727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ce5837-4f6b-4ada-9806-36ef8c668df9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:53.123996 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ce5837-4f6b-4ada-9806-36ef8c668df9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:53.133170 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ce5837-4f6b-4ada-9806-36ef8c668df9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:53.134599 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ce5837-4f6b-4ada-9806-36ef8c668df9 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:53.145014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1067/2174] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:53.501952 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafe045a-aa59-4c2a-b1ed-fd63d9989d2b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:53.502741 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-bafe045a-aa59-4c2a-b1ed-fd63d9989d2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:53.516081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafe045a-aa59-4c2a-b1ed-fd63d9989d2b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:53.520929 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35d0510-0180-4689-b4bf-0a66c0a993ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:53.521717 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35d0510-0180-4689-b4bf-0a66c0a993ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:53.521998 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35d0510-0180-4689-b4bf-0a66c0a993ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:53.522263 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35d0510-0180-4689-b4bf-0a66c0a993ab admin admin] PATH_INFO: `/v3/projects/30ceb8cab7454065b963967e35dde6e2` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:53.523443 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35d0510-0180-4689-b4bf-0a66c0a993ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=30ceb8cab7454065b963967e35dde6e2)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:53.525441 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35d0510-0180-4689-b4bf-0a66c0a993ab admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:53.617827 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-f35d0510-0180-4689-b4bf-0a66c0a993ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '30ceb8cab7454065b963967e35dde6e2', 'request_id': 'req-f35d0510-0180-4689-b4bf-0a66c0a993ab'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:53.628597 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-f35d0510-0180-4689-b4bf-0a66c0a993ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '30ceb8cab7454065b963967e35dde6e2', 'request_id': 'req-f35d0510-0180-4689-b4bf-0a66c0a993ab'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:53.636560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-f35d0510-0180-4689-b4bf-0a66c0a993ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '30ceb8cab7454065b963967e35dde6e2', 'request_id': 'req-f35d0510-0180-4689-b4bf-0a66c0a993ab'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:53.648752 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-f35d0510-0180-4689-b4bf-0a66c0a993ab admin admin] The token cache is being invalidate because project 30ceb8cab7454065b963967e35dde6e2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:03:53.649111 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-f35d0510-0180-4689-b4bf-0a66c0a993ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:53.649769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-f35d0510-0180-4689-b4bf-0a66c0a993ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:53.651395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1108/2175] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:03:53 2022] DELETE /identity/v3/projects/30ceb8cab7454065b963967e35dde6e2 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:53.655442 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ce5837-4f6b-4ada-9806-36ef8c668df9 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:53.665209 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ce5837-4f6b-4ada-9806-36ef8c668df9 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:53.729477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259696a0-7deb-4a2e-bd4b-b5540b636f24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:53.730216 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259696a0-7deb-4a2e-bd4b-b5540b636f24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:53.730712 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259696a0-7deb-4a2e-bd4b-b5540b636f24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:53.731616 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259696a0-7deb-4a2e-bd4b-b5540b636f24 admin admin] PATH_INFO: `/v3/users/f49398aeac73483592cdffa25bfd4613` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:53.732801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259696a0-7deb-4a2e-bd4b-b5540b636f24 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f49398aeac73483592cdffa25bfd4613)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:53.734743 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259696a0-7deb-4a2e-bd4b-b5540b636f24 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:53.786334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-259696a0-7deb-4a2e-bd4b-b5540b636f24 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f49398aeac73483592cdffa25bfd4613'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:53.787553 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-259696a0-7deb-4a2e-bd4b-b5540b636f24 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f49398aeac73483592cdffa25bfd4613'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:53.790126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:53.796774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-259696a0-7deb-4a2e-bd4b-b5540b636f24 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f49398aeac73483592cdffa25bfd4613'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:53.797819 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-259696a0-7deb-4a2e-bd4b-b5540b636f24 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f49398aeac73483592cdffa25bfd4613'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:53.814067 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-259696a0-7deb-4a2e-bd4b-b5540b636f24 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f49398aeac73483592cdffa25bfd4613'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:53.828881 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1068/2176] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:03:53 2022] DELETE /identity/v3/users/f49398aeac73483592cdffa25bfd4613 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:53.931833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca95a374-603c-4f0a-8a80-54138046ca9b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:53.932436 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca95a374-603c-4f0a-8a80-54138046ca9b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:53.932877 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca95a374-603c-4f0a-8a80-54138046ca9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:53.956836 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-ca95a374-603c-4f0a-8a80-54138046ca9b None None] MFA Rules not processed for user `33b136afbf8a4d98a9d62053b68707e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:53.958320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-ca95a374-603c-4f0a-8a80-54138046ca9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:53.996331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:54.057894 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1109/2177] 10.222.0.25 () {58 vars in 967 bytes} [Sun Oct 16 01:03:53 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:54.067091 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259696a0-7deb-4a2e-bd4b-b5540b636f24 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:54.069156 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-259696a0-7deb-4a2e-bd4b-b5540b636f24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:54.151790 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259696a0-7deb-4a2e-bd4b-b5540b636f24 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:54.157278 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f471d375-93b1-462d-b7f0-301e2b5913d0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de64d498da78448394b0d4d1851a2789', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:54.157931 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f471d375-93b1-462d-b7f0-301e2b5913d0 service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:54.158250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f471d375-93b1-462d-b7f0-301e2b5913d0 service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:54.158568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f471d375-93b1-462d-b7f0-301e2b5913d0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:54.168362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f471d375-93b1-462d-b7f0-301e2b5913d0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:54.169310 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca95a374-603c-4f0a-8a80-54138046ca9b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:54.170059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f471d375-93b1-462d-b7f0-301e2b5913d0 service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:54.171161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-ca95a374-603c-4f0a-8a80-54138046ca9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:54.179812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1069/2178] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 16 01:03:54 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 16 01:03:54.263244 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca95a374-603c-4f0a-8a80-54138046ca9b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:54.268369 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825b55e2-59bd-4a4c-b948-2498232d4d6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:54.269017 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825b55e2-59bd-4a4c-b948-2498232d4d6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:54.269299 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825b55e2-59bd-4a4c-b948-2498232d4d6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:54.269570 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825b55e2-59bd-4a4c-b948-2498232d4d6a admin admin] PATH_INFO: `/v3/projects/d65efccc51b242328695e268c0200498` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:54.270886 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825b55e2-59bd-4a4c-b948-2498232d4d6a admin admin] RBAC: Authorizing `identity:delete_project(project_id=d65efccc51b242328695e268c0200498)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:54.274359 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825b55e2-59bd-4a4c-b948-2498232d4d6a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:54.331473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-825b55e2-59bd-4a4c-b948-2498232d4d6a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd65efccc51b242328695e268c0200498', 'request_id': 'req-825b55e2-59bd-4a4c-b948-2498232d4d6a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:54.342841 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-825b55e2-59bd-4a4c-b948-2498232d4d6a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd65efccc51b242328695e268c0200498', 'request_id': 'req-825b55e2-59bd-4a4c-b948-2498232d4d6a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:54.352716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-825b55e2-59bd-4a4c-b948-2498232d4d6a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd65efccc51b242328695e268c0200498', 'request_id': 'req-825b55e2-59bd-4a4c-b948-2498232d4d6a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:54.376943 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-825b55e2-59bd-4a4c-b948-2498232d4d6a admin admin] The token cache is being invalidate because project d65efccc51b242328695e268c0200498 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:03:54.377203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-825b55e2-59bd-4a4c-b948-2498232d4d6a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:54.378104 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-825b55e2-59bd-4a4c-b948-2498232d4d6a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:54.380118 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1110/2179] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:03:54 2022] DELETE /identity/v3/projects/d65efccc51b242328695e268c0200498 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:54.388140 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2164b9fc-b9ee-4387-9801-65fba0bc3974 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:54.388350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2164b9fc-b9ee-4387-9801-65fba0bc3974 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:54.388618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2164b9fc-b9ee-4387-9801-65fba0bc3974 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:54.415542 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2164b9fc-b9ee-4387-9801-65fba0bc3974 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:54.417153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2164b9fc-b9ee-4387-9801-65fba0bc3974 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:54.545811 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1070/2180] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:54.548142 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8c234f-4606-4473-a28f-838622bc52a9 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:54.548607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8c234f-4606-4473-a28f-838622bc52a9 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:54.548998 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8c234f-4606-4473-a28f-838622bc52a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:54.580435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-ae8c234f-4606-4473-a28f-838622bc52a9 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:54.586230 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-ae8c234f-4606-4473-a28f-838622bc52a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:54.599852 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1111/2181] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:54.601221 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2164b9fc-b9ee-4387-9801-65fba0bc3974 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:54.612894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2164b9fc-b9ee-4387-9801-65fba0bc3974 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:54.623081 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05f9f24-fca0-407c-aea3-cfd38fa34fa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:54.624338 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05f9f24-fca0-407c-aea3-cfd38fa34fa4 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:54.624753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05f9f24-fca0-407c-aea3-cfd38fa34fa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:54.625120 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05f9f24-fca0-407c-aea3-cfd38fa34fa4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:54.625869 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05f9f24-fca0-407c-aea3-cfd38fa34fa4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:03:54.626269 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05f9f24-fca0-407c-aea3-cfd38fa34fa4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:54.628090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05f9f24-fca0-407c-aea3-cfd38fa34fa4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:54.648576 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1071/2182] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:03:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:03:54.665756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8c234f-4606-4473-a28f-838622bc52a9 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:54.676264 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8c234f-4606-4473-a28f-838622bc52a9 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:54.684320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b009f6-35e8-401d-8fa2-a2c152c0cf2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:54.684938 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b009f6-35e8-401d-8fa2-a2c152c0cf2b admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:54.685214 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b009f6-35e8-401d-8fa2-a2c152c0cf2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:54.685483 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b009f6-35e8-401d-8fa2-a2c152c0cf2b admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:54.686255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b009f6-35e8-401d-8fa2-a2c152c0cf2b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:54.688671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b009f6-35e8-401d-8fa2-a2c152c0cf2b admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:54.723891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1112/2183] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:03:54 2022] POST /identity/v3/projects => generated 379 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:03:54.725042 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05f9f24-fca0-407c-aea3-cfd38fa34fa4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:54.737568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05f9f24-fca0-407c-aea3-cfd38fa34fa4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:54.784039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:DELETE=3,keystone:SELECT=97 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:54.800774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c4ef47-ea0c-4fab-abb5-90455ea9eb02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:54.801429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c4ef47-ea0c-4fab-abb5-90455ea9eb02 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:54.801556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c4ef47-ea0c-4fab-abb5-90455ea9eb02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:54.801831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c4ef47-ea0c-4fab-abb5-90455ea9eb02 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:54.802519 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c4ef47-ea0c-4fab-abb5-90455ea9eb02 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:54.804584 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c4ef47-ea0c-4fab-abb5-90455ea9eb02 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:54.830518 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1072/2184] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:03:54 2022] POST /identity/v3/users => generated 290 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 01:03:54.832083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b009f6-35e8-401d-8fa2-a2c152c0cf2b admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:54.843648 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b009f6-35e8-401d-8fa2-a2c152c0cf2b admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:54.848281 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c11f56f-5a5c-47c3-bd27-28f15500d09f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:54.848880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c11f56f-5a5c-47c3-bd27-28f15500d09f admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:54.849160 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c11f56f-5a5c-47c3-bd27-28f15500d09f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:54.849574 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c11f56f-5a5c-47c3-bd27-28f15500d09f admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:54.850165 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c11f56f-5a5c-47c3-bd27-28f15500d09f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:54.851808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c11f56f-5a5c-47c3-bd27-28f15500d09f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:54.862093 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1113/2185] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:03:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:03:54.866399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c4ef47-ea0c-4fab-abb5-90455ea9eb02 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:54.876472 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c4ef47-ea0c-4fab-abb5-90455ea9eb02 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:54.881934 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113647c2-29dc-48bd-b4a0-39b00ad9a9c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:54.882514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113647c2-29dc-48bd-b4a0-39b00ad9a9c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:54.882788 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113647c2-29dc-48bd-b4a0-39b00ad9a9c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:54.883092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113647c2-29dc-48bd-b4a0-39b00ad9a9c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:54.883734 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113647c2-29dc-48bd-b4a0-39b00ad9a9c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:54.885385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113647c2-29dc-48bd-b4a0-39b00ad9a9c9 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:54.896690 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1073/2186] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:03:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:03:54.901016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c11f56f-5a5c-47c3-bd27-28f15500d09f admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:54.911212 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c11f56f-5a5c-47c3-bd27-28f15500d09f admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:54.916117 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6be024-e485-4ce2-80ad-d0fad6e00f5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:54.916784 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6be024-e485-4ce2-80ad-d0fad6e00f5b admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:54.917068 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6be024-e485-4ce2-80ad-d0fad6e00f5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:54.917473 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6be024-e485-4ce2-80ad-d0fad6e00f5b admin admin] PATH_INFO: `/v3/projects/0a5be2f1df3f4739964817caa21ea82f/users/a0089f7a4b4a4c80bde55ca66f88ea8b/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:54.934244 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6be024-e485-4ce2-80ad-d0fad6e00f5b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a5be2f1df3f4739964817caa21ea82f, user_id=a0089f7a4b4a4c80bde55ca66f88ea8b, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:54.936991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6be024-e485-4ce2-80ad-d0fad6e00f5b admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:54.952283 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1114/2187] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:03:54 2022] PUT /identity/v3/projects/0a5be2f1df3f4739964817caa21ea82f/users/a0089f7a4b4a4c80bde55ca66f88ea8b/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:03:54.963107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc118c5d-df10-4741-a674-c1f339aa8f3e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:54.963394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc118c5d-df10-4741-a674-c1f339aa8f3e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:54.963664 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc118c5d-df10-4741-a674-c1f339aa8f3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:54.992911 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-fc118c5d-df10-4741-a674-c1f339aa8f3e None None] MFA Rules not processed for user `a0089f7a4b4a4c80bde55ca66f88ea8b`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:54.994130 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fc118c5d-df10-4741-a674-c1f339aa8f3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:55.062688 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1074/2188] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:54 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:55.067060 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6be024-e485-4ce2-80ad-d0fad6e00f5b admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:55.073260 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:55.151507 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6be024-e485-4ce2-80ad-d0fad6e00f5b admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:55.156769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f9ded5-dfa2-478c-8db9-ad5f455d5761 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:55.157394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f9ded5-dfa2-478c-8db9-ad5f455d5761 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:55.157686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f9ded5-dfa2-478c-8db9-ad5f455d5761 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:55.158935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f9ded5-dfa2-478c-8db9-ad5f455d5761 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:55.158935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f9ded5-dfa2-478c-8db9-ad5f455d5761 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:55.160617 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f9ded5-dfa2-478c-8db9-ad5f455d5761 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:55.203592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1115/2189] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:03:55 2022] POST /identity/v3/projects => generated 381 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:03:55.204786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc118c5d-df10-4741-a674-c1f339aa8f3e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:55.218139 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc118c5d-df10-4741-a674-c1f339aa8f3e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:55.278277 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0630820-2b77-4626-8099-06485333305e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:55.278930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0630820-2b77-4626-8099-06485333305e admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:55.279220 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0630820-2b77-4626-8099-06485333305e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:55.279493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0630820-2b77-4626-8099-06485333305e admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:55.280140 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0630820-2b77-4626-8099-06485333305e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:55.282109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0630820-2b77-4626-8099-06485333305e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:55.304350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1075/2190] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:03:55 2022] POST /identity/v3/users => generated 297 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 01:03:55.308545 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f9ded5-dfa2-478c-8db9-ad5f455d5761 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:55.319551 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f9ded5-dfa2-478c-8db9-ad5f455d5761 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:55.325449 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1103350b-ca68-42dd-85d3-035061e5b596 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:55.326742 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1103350b-ca68-42dd-85d3-035061e5b596 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:55.327126 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1103350b-ca68-42dd-85d3-035061e5b596 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:55.327408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1103350b-ca68-42dd-85d3-035061e5b596 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:55.328065 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1103350b-ca68-42dd-85d3-035061e5b596 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:55.329716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1103350b-ca68-42dd-85d3-035061e5b596 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:55.343576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1116/2191] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:03:55 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:03:55.347825 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0630820-2b77-4626-8099-06485333305e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:55.357970 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0630820-2b77-4626-8099-06485333305e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:55.362837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c298fc-39e2-432d-b737-36f3e5f4aaf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:55.363630 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c298fc-39e2-432d-b737-36f3e5f4aaf9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:55.363906 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c298fc-39e2-432d-b737-36f3e5f4aaf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:55.364181 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c298fc-39e2-432d-b737-36f3e5f4aaf9 admin admin] PATH_INFO: `/v3/projects/3cff058f536f4a10ab740d7f2faf599d/users/4612e8457f5e41c796fc43a20aa2f2b2/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:55.379420 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c298fc-39e2-432d-b737-36f3e5f4aaf9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3cff058f536f4a10ab740d7f2faf599d, user_id=4612e8457f5e41c796fc43a20aa2f2b2, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:55.382726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c298fc-39e2-432d-b737-36f3e5f4aaf9 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:55.396093 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1076/2192] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:03:55 2022] PUT /identity/v3/projects/3cff058f536f4a10ab740d7f2faf599d/users/4612e8457f5e41c796fc43a20aa2f2b2/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:55.401121 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1103350b-ca68-42dd-85d3-035061e5b596 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:55.411389 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1103350b-ca68-42dd-85d3-035061e5b596 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:55.469749 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5299c5-720c-4f52-a2d7-51c315d9f2ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:55.470367 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5299c5-720c-4f52-a2d7-51c315d9f2ee admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:55.470647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5299c5-720c-4f52-a2d7-51c315d9f2ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:55.470952 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5299c5-720c-4f52-a2d7-51c315d9f2ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:55.471592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5299c5-720c-4f52-a2d7-51c315d9f2ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:55.473186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5299c5-720c-4f52-a2d7-51c315d9f2ee admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:55.484092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1117/2193] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:03:55 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:03:55.488323 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c298fc-39e2-432d-b737-36f3e5f4aaf9 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:55.498972 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c298fc-39e2-432d-b737-36f3e5f4aaf9 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:55.503632 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9693ce3-575d-4244-aade-ca560d9dde8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:55.504282 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9693ce3-575d-4244-aade-ca560d9dde8b admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:55.504568 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9693ce3-575d-4244-aade-ca560d9dde8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:55.504844 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9693ce3-575d-4244-aade-ca560d9dde8b admin admin] PATH_INFO: `/v3/projects/3cff058f536f4a10ab740d7f2faf599d/users/4612e8457f5e41c796fc43a20aa2f2b2/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:55.507406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9693ce3-575d-4244-aade-ca560d9dde8b admin admin] RBAC: Authorizing `identity:create_grant(project_id=3cff058f536f4a10ab740d7f2faf599d, user_id=4612e8457f5e41c796fc43a20aa2f2b2, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:55.510071 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9693ce3-575d-4244-aade-ca560d9dde8b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:55.520309 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1077/2194] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:03:55 2022] PUT /identity/v3/projects/3cff058f536f4a10ab740d7f2faf599d/users/4612e8457f5e41c796fc43a20aa2f2b2/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:55.532323 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabcd48c-2745-44eb-96e4-e45c2dfd1c90 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:55.532610 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabcd48c-2745-44eb-96e4-e45c2dfd1c90 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:55.532885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabcd48c-2745-44eb-96e4-e45c2dfd1c90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:55.565914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-fabcd48c-2745-44eb-96e4-e45c2dfd1c90 None None] MFA Rules not processed for user `4612e8457f5e41c796fc43a20aa2f2b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:55.566688 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-fabcd48c-2745-44eb-96e4-e45c2dfd1c90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:55.632470 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:55.638877 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1118/2195] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:55 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:55.651327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9693ce3-575d-4244-aade-ca560d9dde8b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:55.653076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f9693ce3-575d-4244-aade-ca560d9dde8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:55.659672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:55.723079 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9693ce3-575d-4244-aade-ca560d9dde8b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:55.726999 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b3b7e0-02ec-49ee-b928-43eb8e211f3a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:55.727886 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b3b7e0-02ec-49ee-b928-43eb8e211f3a service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:55.728248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b3b7e0-02ec-49ee-b928-43eb8e211f3a service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:55.728594 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b3b7e0-02ec-49ee-b928-43eb8e211f3a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:55.740349 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b3b7e0-02ec-49ee-b928-43eb8e211f3a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:55.742037 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b3b7e0-02ec-49ee-b928-43eb8e211f3a service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:55.752726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1078/2196] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:55 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:55.772490 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce50b1b-ded6-4ffe-a427-3a3697537094 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:55.773479 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce50b1b-ded6-4ffe-a427-3a3697537094 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:55.773755 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce50b1b-ded6-4ffe-a427-3a3697537094 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:55.799932 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-8ce50b1b-ded6-4ffe-a427-3a3697537094 None None] MFA Rules not processed for user `33b136afbf8a4d98a9d62053b68707e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:55.801173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-8ce50b1b-ded6-4ffe-a427-3a3697537094 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:55.868229 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1119/2197] 10.222.0.25 () {58 vars in 967 bytes} [Sun Oct 16 01:03:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:55.890437 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b3b7e0-02ec-49ee-b928-43eb8e211f3a service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:55.892409 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f5b3b7e0-02ec-49ee-b928-43eb8e211f3a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:55.970745 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b3b7e0-02ec-49ee-b928-43eb8e211f3a service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:55.977313 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec10d44f-9803-42f7-9fba-5f430e4fdba5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de64d498da78448394b0d4d1851a2789', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:55.978049 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec10d44f-9803-42f7-9fba-5f430e4fdba5 service swift] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:55.978405 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec10d44f-9803-42f7-9fba-5f430e4fdba5 service swift] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:55.978821 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec10d44f-9803-42f7-9fba-5f430e4fdba5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:55.990577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec10d44f-9803-42f7-9fba-5f430e4fdba5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:55.992489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec10d44f-9803-42f7-9fba-5f430e4fdba5 service swift] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:56.002492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1079/2198] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 16 01:03:55 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 16 01:03:56.134170 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce50b1b-ded6-4ffe-a427-3a3697537094 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:56.149886 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce50b1b-ded6-4ffe-a427-3a3697537094 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:56.154703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1cce48-2575-466c-a7ba-2531e5203e44 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4612e8457f5e41c796fc43a20aa2f2b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cff058f536f4a10ab740d7f2faf599d', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:56.155311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1cce48-2575-466c-a7ba-2531e5203e44 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:56.155618 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1cce48-2575-466c-a7ba-2531e5203e44 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:56.155822 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1cce48-2575-466c-a7ba-2531e5203e44 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:56.156672 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1cce48-2575-466c-a7ba-2531e5203e44 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:03:56.157159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1cce48-2575-466c-a7ba-2531e5203e44 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:56.159272 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1cce48-2575-466c-a7ba-2531e5203e44 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:56.176097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1120/2199] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:03:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:03:56.180591 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec10d44f-9803-42f7-9fba-5f430e4fdba5 service swift] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:56.194017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec10d44f-9803-42f7-9fba-5f430e4fdba5 service swift] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:56.199921 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a946503d-d839-4a25-90d3-4e6edde21b83 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4612e8457f5e41c796fc43a20aa2f2b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cff058f536f4a10ab740d7f2faf599d', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:56.200522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a946503d-d839-4a25-90d3-4e6edde21b83 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:56.200874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a946503d-d839-4a25-90d3-4e6edde21b83 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:56.201322 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a946503d-d839-4a25-90d3-4e6edde21b83 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:56.201929 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a946503d-d839-4a25-90d3-4e6edde21b83 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:56.204071 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a946503d-d839-4a25-90d3-4e6edde21b83 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:56.245313 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1080/2200] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:03:56 2022] POST /identity/v3/projects => generated 354 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:03:56.249166 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1cce48-2575-466c-a7ba-2531e5203e44 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:56.260654 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1cce48-2575-466c-a7ba-2531e5203e44 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:56.321342 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4264b1b0-982d-4f51-9f96-5bd2cc679525 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4612e8457f5e41c796fc43a20aa2f2b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cff058f536f4a10ab740d7f2faf599d', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:56.322211 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4264b1b0-982d-4f51-9f96-5bd2cc679525 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:56.322497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4264b1b0-982d-4f51-9f96-5bd2cc679525 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:56.322772 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4264b1b0-982d-4f51-9f96-5bd2cc679525 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:56.323452 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4264b1b0-982d-4f51-9f96-5bd2cc679525 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:03:56.323761 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4264b1b0-982d-4f51-9f96-5bd2cc679525 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:56.325368 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4264b1b0-982d-4f51-9f96-5bd2cc679525 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:56.339745 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1121/2201] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:03:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:03:56.383749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a946503d-d839-4a25-90d3-4e6edde21b83 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:56.395321 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a946503d-d839-4a25-90d3-4e6edde21b83 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:56.408288 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd59720f-2c77-48f7-93d4-84da8bd40574 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4612e8457f5e41c796fc43a20aa2f2b2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '3cff058f536f4a10ab740d7f2faf599d', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:56.408949 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd59720f-2c77-48f7-93d4-84da8bd40574 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:56.409282 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd59720f-2c77-48f7-93d4-84da8bd40574 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:56.409550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd59720f-2c77-48f7-93d4-84da8bd40574 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] PATH_INFO: `/v3/projects/3b9bc9a5c5754f0c99be02fe25957140` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:56.410735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd59720f-2c77-48f7-93d4-84da8bd40574 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] RBAC: Authorizing `identity:delete_project(project_id=3b9bc9a5c5754f0c99be02fe25957140)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:56.413753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd59720f-2c77-48f7-93d4-84da8bd40574 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:56.473686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cd59720f-2c77-48f7-93d4-84da8bd40574 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3b9bc9a5c5754f0c99be02fe25957140', 'request_id': 'req-cd59720f-2c77-48f7-93d4-84da8bd40574'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:56.484010 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cd59720f-2c77-48f7-93d4-84da8bd40574 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3b9bc9a5c5754f0c99be02fe25957140', 'request_id': 'req-cd59720f-2c77-48f7-93d4-84da8bd40574'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:56.492766 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cd59720f-2c77-48f7-93d4-84da8bd40574 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3b9bc9a5c5754f0c99be02fe25957140', 'request_id': 'req-cd59720f-2c77-48f7-93d4-84da8bd40574'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:56.499424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:56.505182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cd59720f-2c77-48f7-93d4-84da8bd40574 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] The token cache is being invalidate because project 3b9bc9a5c5754f0c99be02fe25957140 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:03:56.505892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cd59720f-2c77-48f7-93d4-84da8bd40574 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:56.506967 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cd59720f-2c77-48f7-93d4-84da8bd40574 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:56.509331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1081/2202] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:03:56 2022] DELETE /identity/v3/projects/3b9bc9a5c5754f0c99be02fe25957140 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:56.723134 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4264b1b0-982d-4f51-9f96-5bd2cc679525 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:56.725653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4264b1b0-982d-4f51-9f96-5bd2cc679525 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:56.793623 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4264b1b0-982d-4f51-9f96-5bd2cc679525 tempest-VolumeQuotaClassesTest-1505250754 tempest-VolumeQuotaClassesTest-1505250754-project-admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:56.825874 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:56.857381 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123e6736-078e-475c-9461-d79b062b79b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:56.857948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123e6736-078e-475c-9461-d79b062b79b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:56.858234 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123e6736-078e-475c-9461-d79b062b79b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:56.858481 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123e6736-078e-475c-9461-d79b062b79b4 admin admin] PATH_INFO: `/v3/users/a0089f7a4b4a4c80bde55ca66f88ea8b` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:56.859689 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123e6736-078e-475c-9461-d79b062b79b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a0089f7a4b4a4c80bde55ca66f88ea8b)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:56.861583 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123e6736-078e-475c-9461-d79b062b79b4 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:56.919128 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-123e6736-078e-475c-9461-d79b062b79b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a0089f7a4b4a4c80bde55ca66f88ea8b'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:56.919875 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-123e6736-078e-475c-9461-d79b062b79b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a0089f7a4b4a4c80bde55ca66f88ea8b'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:56.929195 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-123e6736-078e-475c-9461-d79b062b79b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a0089f7a4b4a4c80bde55ca66f88ea8b'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:56.946543 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-123e6736-078e-475c-9461-d79b062b79b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a0089f7a4b4a4c80bde55ca66f88ea8b'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:56.954384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-123e6736-078e-475c-9461-d79b062b79b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a0089f7a4b4a4c80bde55ca66f88ea8b'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:56.961716 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1122/2203] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:03:56 2022] DELETE /identity/v3/users/a0089f7a4b4a4c80bde55ca66f88ea8b => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:56.966268 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402b7cc1-af08-4d16-9211-44f1e88b905e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:56.966920 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402b7cc1-af08-4d16-9211-44f1e88b905e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:56.966920 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402b7cc1-af08-4d16-9211-44f1e88b905e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:56.987792 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-402b7cc1-af08-4d16-9211-44f1e88b905e None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:56.989680 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-402b7cc1-af08-4d16-9211-44f1e88b905e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:57.053060 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1082/2204] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:57.062983 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123e6736-078e-475c-9461-d79b062b79b4 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:57.064433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-123e6736-078e-475c-9461-d79b062b79b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:57.150365 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123e6736-078e-475c-9461-d79b062b79b4 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:57.155794 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4b735d-06a2-4216-912a-8db443bec386 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:57.156526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4b735d-06a2-4216-912a-8db443bec386 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:57.156808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4b735d-06a2-4216-912a-8db443bec386 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:57.157083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4b735d-06a2-4216-912a-8db443bec386 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:57.166732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4b735d-06a2-4216-912a-8db443bec386 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:57.168134 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4b735d-06a2-4216-912a-8db443bec386 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:57.178698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1123/2205] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:03:57.468831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402b7cc1-af08-4d16-9211-44f1e88b905e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:57.470306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-402b7cc1-af08-4d16-9211-44f1e88b905e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:57.486530 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402b7cc1-af08-4d16-9211-44f1e88b905e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:57.494662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766fe357-f141-449f-a8fc-47f25a4c3f66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:57.495600 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766fe357-f141-449f-a8fc-47f25a4c3f66 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:57.496117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766fe357-f141-449f-a8fc-47f25a4c3f66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:57.496605 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766fe357-f141-449f-a8fc-47f25a4c3f66 admin admin] PATH_INFO: `/v3/projects/0a5be2f1df3f4739964817caa21ea82f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:57.498378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766fe357-f141-449f-a8fc-47f25a4c3f66 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a5be2f1df3f4739964817caa21ea82f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:57.500605 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766fe357-f141-449f-a8fc-47f25a4c3f66 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:57.559836 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-766fe357-f141-449f-a8fc-47f25a4c3f66 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a5be2f1df3f4739964817caa21ea82f', 'request_id': 'req-766fe357-f141-449f-a8fc-47f25a4c3f66'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:57.572418 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-766fe357-f141-449f-a8fc-47f25a4c3f66 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a5be2f1df3f4739964817caa21ea82f', 'request_id': 'req-766fe357-f141-449f-a8fc-47f25a4c3f66'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:57.585030 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-766fe357-f141-449f-a8fc-47f25a4c3f66 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a5be2f1df3f4739964817caa21ea82f', 'request_id': 'req-766fe357-f141-449f-a8fc-47f25a4c3f66'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:57.601308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-766fe357-f141-449f-a8fc-47f25a4c3f66 admin admin] The token cache is being invalidate because project 0a5be2f1df3f4739964817caa21ea82f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:03:57.601603 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-766fe357-f141-449f-a8fc-47f25a4c3f66 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:57.602326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-766fe357-f141-449f-a8fc-47f25a4c3f66 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:57.604057 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1083/2206] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:03:57 2022] DELETE /identity/v3/projects/0a5be2f1df3f4739964817caa21ea82f => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:57.608110 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4b735d-06a2-4216-912a-8db443bec386 service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:57.625454 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4b735d-06a2-4216-912a-8db443bec386 service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:57.683425 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:57.695399 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479b7d51-bcdc-4e08-91cc-b97948a2893e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:57.697256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479b7d51-bcdc-4e08-91cc-b97948a2893e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:57.697613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479b7d51-bcdc-4e08-91cc-b97948a2893e admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:57.697997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479b7d51-bcdc-4e08-91cc-b97948a2893e admin admin] PATH_INFO: `/v3/users/4612e8457f5e41c796fc43a20aa2f2b2` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:57.699418 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479b7d51-bcdc-4e08-91cc-b97948a2893e admin admin] RBAC: Authorizing `identity:delete_user(user_id=4612e8457f5e41c796fc43a20aa2f2b2)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:57.702263 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479b7d51-bcdc-4e08-91cc-b97948a2893e admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:57.765338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-479b7d51-bcdc-4e08-91cc-b97948a2893e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4612e8457f5e41c796fc43a20aa2f2b2'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:57.766170 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-479b7d51-bcdc-4e08-91cc-b97948a2893e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4612e8457f5e41c796fc43a20aa2f2b2'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:57.774689 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-479b7d51-bcdc-4e08-91cc-b97948a2893e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4612e8457f5e41c796fc43a20aa2f2b2'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:57.792933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-479b7d51-bcdc-4e08-91cc-b97948a2893e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4612e8457f5e41c796fc43a20aa2f2b2'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:57.804374 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-479b7d51-bcdc-4e08-91cc-b97948a2893e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4612e8457f5e41c796fc43a20aa2f2b2'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:57.811548 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1124/2207] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:03:57 2022] DELETE /identity/v3/users/4612e8457f5e41c796fc43a20aa2f2b2 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:03:58.261251 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766fe357-f141-449f-a8fc-47f25a4c3f66 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:58.262905 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-766fe357-f141-449f-a8fc-47f25a4c3f66 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:58.286449 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766fe357-f141-449f-a8fc-47f25a4c3f66 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:58.384310 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba811eb-b394-4914-8317-6159c59c031c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:58.384958 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba811eb-b394-4914-8317-6159c59c031c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:58.385234 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba811eb-b394-4914-8317-6159c59c031c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:58.385504 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba811eb-b394-4914-8317-6159c59c031c admin admin] PATH_INFO: `/v3/projects/3cff058f536f4a10ab740d7f2faf599d` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:58.386850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba811eb-b394-4914-8317-6159c59c031c admin admin] RBAC: Authorizing `identity:delete_project(project_id=3cff058f536f4a10ab740d7f2faf599d)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:58.388960 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba811eb-b394-4914-8317-6159c59c031c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:58.436591 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:58.478058 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4ba811eb-b394-4914-8317-6159c59c031c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3cff058f536f4a10ab740d7f2faf599d', 'request_id': 'req-4ba811eb-b394-4914-8317-6159c59c031c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:58.506524 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4ba811eb-b394-4914-8317-6159c59c031c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3cff058f536f4a10ab740d7f2faf599d', 'request_id': 'req-4ba811eb-b394-4914-8317-6159c59c031c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:58.522360 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4ba811eb-b394-4914-8317-6159c59c031c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3cff058f536f4a10ab740d7f2faf599d', 'request_id': 'req-4ba811eb-b394-4914-8317-6159c59c031c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:58.536394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4ba811eb-b394-4914-8317-6159c59c031c admin admin] The token cache is being invalidate because project 3cff058f536f4a10ab740d7f2faf599d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:03:58.536995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4ba811eb-b394-4914-8317-6159c59c031c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:58.537893 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4ba811eb-b394-4914-8317-6159c59c031c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:03:58.540469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1084/2208] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:03:58 2022] DELETE /identity/v3/projects/3cff058f536f4a10ab740d7f2faf599d => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:58.546536 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d1b60a-c24e-4cad-b015-12fc6b320c2f None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:58.547075 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d1b60a-c24e-4cad-b015-12fc6b320c2f None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:58.547075 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d1b60a-c24e-4cad-b015-12fc6b320c2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:58.574649 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-59d1b60a-c24e-4cad-b015-12fc6b320c2f None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:58.575660 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-59d1b60a-c24e-4cad-b015-12fc6b320c2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:58.655992 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1125/2209] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:58.664173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b65d11-3bec-4fed-8185-0db456152526 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:58.664464 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b65d11-3bec-4fed-8185-0db456152526 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:58.664728 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b65d11-3bec-4fed-8185-0db456152526 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:58.688356 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f8b65d11-3bec-4fed-8185-0db456152526 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:58.690002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f8b65d11-3bec-4fed-8185-0db456152526 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:58.709775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1085/2210] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:58.714815 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d1b60a-c24e-4cad-b015-12fc6b320c2f None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:58.728879 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d1b60a-c24e-4cad-b015-12fc6b320c2f None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:58.735691 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32f52ef-7533-40fd-8139-508860bfedbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:58.736683 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32f52ef-7533-40fd-8139-508860bfedbf admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:58.737155 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32f52ef-7533-40fd-8139-508860bfedbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:58.737635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32f52ef-7533-40fd-8139-508860bfedbf admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:58.738494 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32f52ef-7533-40fd-8139-508860bfedbf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:03:58.739033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32f52ef-7533-40fd-8139-508860bfedbf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:58.740994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32f52ef-7533-40fd-8139-508860bfedbf admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:58.757977 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1126/2211] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:03:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 01:03:58.762245 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b65d11-3bec-4fed-8185-0db456152526 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:58.777224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b65d11-3bec-4fed-8185-0db456152526 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:58.784235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc4a9b8-f903-4d96-a17c-1c7f7dc04735 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:58.784851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc4a9b8-f903-4d96-a17c-1c7f7dc04735 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:58.785136 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc4a9b8-f903-4d96-a17c-1c7f7dc04735 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:58.785459 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc4a9b8-f903-4d96-a17c-1c7f7dc04735 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:58.786128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc4a9b8-f903-4d96-a17c-1c7f7dc04735 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:58.789495 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc4a9b8-f903-4d96-a17c-1c7f7dc04735 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:58.830921 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1086/2212] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:03:58 2022] POST /identity/v3/projects => generated 401 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 01:03:58.836873 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32f52ef-7533-40fd-8139-508860bfedbf admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:58.856928 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32f52ef-7533-40fd-8139-508860bfedbf admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:58.919592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d2babf-11ea-46cc-b14d-32a45fa70e11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:58.920215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d2babf-11ea-46cc-b14d-32a45fa70e11 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:58.920499 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d2babf-11ea-46cc-b14d-32a45fa70e11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:58.920773 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d2babf-11ea-46cc-b14d-32a45fa70e11 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:58.921416 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d2babf-11ea-46cc-b14d-32a45fa70e11 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:58.923394 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d2babf-11ea-46cc-b14d-32a45fa70e11 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:58.932129 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:58.949802 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1127/2213] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:03:58 2022] POST /identity/v3/users => generated 301 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:03:58.954210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc4a9b8-f903-4d96-a17c-1c7f7dc04735 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:58.965114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc4a9b8-f903-4d96-a17c-1c7f7dc04735 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:58.970128 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3e8779-f55f-44c3-b286-232128113c10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:58.970742 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3e8779-f55f-44c3-b286-232128113c10 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:58.971056 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3e8779-f55f-44c3-b286-232128113c10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:58.971332 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3e8779-f55f-44c3-b286-232128113c10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:58.971984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3e8779-f55f-44c3-b286-232128113c10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:58.973622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3e8779-f55f-44c3-b286-232128113c10 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:58.987905 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1087/2214] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:03:58 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:03:58.989128 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d2babf-11ea-46cc-b14d-32a45fa70e11 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:59.000149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d2babf-11ea-46cc-b14d-32a45fa70e11 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:59.004768 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a06e326-de79-4d2d-90a1-b065078ce2c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:59.005524 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a06e326-de79-4d2d-90a1-b065078ce2c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:59.005989 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a06e326-de79-4d2d-90a1-b065078ce2c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:59.006401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a06e326-de79-4d2d-90a1-b065078ce2c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:59.007085 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a06e326-de79-4d2d-90a1-b065078ce2c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:59.008762 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a06e326-de79-4d2d-90a1-b065078ce2c9 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:59.030128 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1128/2215] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:03:58 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:03:59.030758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3e8779-f55f-44c3-b286-232128113c10 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:59.041983 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3e8779-f55f-44c3-b286-232128113c10 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:59.046753 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d6a5f3-792f-42b1-b025-571fcb56976f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:59.047434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d6a5f3-792f-42b1-b025-571fcb56976f admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:59.047715 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d6a5f3-792f-42b1-b025-571fcb56976f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:59.047992 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d6a5f3-792f-42b1-b025-571fcb56976f admin admin] PATH_INFO: `/v3/projects/5fe7780fa7f141e89f06bc9e9cbb7b78/users/25165bf99f82450eabf4a91bbd3b8641/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:59.064635 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d6a5f3-792f-42b1-b025-571fcb56976f admin admin] RBAC: Authorizing `identity:create_grant(project_id=5fe7780fa7f141e89f06bc9e9cbb7b78, user_id=25165bf99f82450eabf4a91bbd3b8641, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:59.067487 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d6a5f3-792f-42b1-b025-571fcb56976f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:59.081751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1088/2216] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:03:59 2022] PUT /identity/v3/projects/5fe7780fa7f141e89f06bc9e9cbb7b78/users/25165bf99f82450eabf4a91bbd3b8641/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:59.093518 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc25f72-7203-4415-bf79-287803690ff4 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:59.093842 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc25f72-7203-4415-bf79-287803690ff4 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:59.094113 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc25f72-7203-4415-bf79-287803690ff4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:59.124203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-6dc25f72-7203-4415-bf79-287803690ff4 None None] MFA Rules not processed for user `25165bf99f82450eabf4a91bbd3b8641`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:59.125429 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6dc25f72-7203-4415-bf79-287803690ff4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:59.201323 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1129/2217] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:59 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 01:03:59.205531 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d6a5f3-792f-42b1-b025-571fcb56976f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:59.279134 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d6a5f3-792f-42b1-b025-571fcb56976f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:59.284061 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9706aa8e-be5d-4513-96ce-ca0623bb1f2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:59.284699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9706aa8e-be5d-4513-96ce-ca0623bb1f2e admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:59.284983 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9706aa8e-be5d-4513-96ce-ca0623bb1f2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:59.285259 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9706aa8e-be5d-4513-96ce-ca0623bb1f2e admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:59.285891 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9706aa8e-be5d-4513-96ce-ca0623bb1f2e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:59.288719 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9706aa8e-be5d-4513-96ce-ca0623bb1f2e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:59.327670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1089/2218] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:03:59 2022] POST /identity/v3/projects => generated 401 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:03:59.330697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc25f72-7203-4415-bf79-287803690ff4 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:59.345028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc25f72-7203-4415-bf79-287803690ff4 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:59.417788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b941e785-6f2c-42e7-b612-8cd43451630f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:59.418448 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b941e785-6f2c-42e7-b612-8cd43451630f admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:59.418687 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b941e785-6f2c-42e7-b612-8cd43451630f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:59.419472 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b941e785-6f2c-42e7-b612-8cd43451630f admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:59.419685 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b941e785-6f2c-42e7-b612-8cd43451630f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:59.422328 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b941e785-6f2c-42e7-b612-8cd43451630f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:59.455019 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1130/2219] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:03:59 2022] POST /identity/v3/users => generated 307 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:03:59.457466 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9706aa8e-be5d-4513-96ce-ca0623bb1f2e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:59.469739 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9706aa8e-be5d-4513-96ce-ca0623bb1f2e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:59.475471 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cffef3-3e8d-4cfb-bbd9-33d7bd50db67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:59.476089 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cffef3-3e8d-4cfb-bbd9-33d7bd50db67 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:59.476373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cffef3-3e8d-4cfb-bbd9-33d7bd50db67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:59.476651 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cffef3-3e8d-4cfb-bbd9-33d7bd50db67 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:59.477305 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cffef3-3e8d-4cfb-bbd9-33d7bd50db67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:59.478999 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cffef3-3e8d-4cfb-bbd9-33d7bd50db67 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:59.490996 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1090/2220] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:03:59 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:03:59.495156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b941e785-6f2c-42e7-b612-8cd43451630f admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:59.505781 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b941e785-6f2c-42e7-b612-8cd43451630f admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:59.511120 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b55f107-edd3-4eb2-8cf5-e4e52b561f5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:59.511866 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b55f107-edd3-4eb2-8cf5-e4e52b561f5c admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:59.512215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b55f107-edd3-4eb2-8cf5-e4e52b561f5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:59.512600 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b55f107-edd3-4eb2-8cf5-e4e52b561f5c admin admin] PATH_INFO: `/v3/projects/68b1319516144f958fd2daf6fc42b42e/users/9eec3b89e5934cad930bc5fb89aa91c8/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:59.530924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b55f107-edd3-4eb2-8cf5-e4e52b561f5c admin admin] RBAC: Authorizing `identity:create_grant(project_id=68b1319516144f958fd2daf6fc42b42e, user_id=9eec3b89e5934cad930bc5fb89aa91c8, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:59.533874 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b55f107-edd3-4eb2-8cf5-e4e52b561f5c admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:59.556363 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1131/2221] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:03:59 2022] PUT /identity/v3/projects/68b1319516144f958fd2daf6fc42b42e/users/9eec3b89e5934cad930bc5fb89aa91c8/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:03:59.558342 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cffef3-3e8d-4cfb-bbd9-33d7bd50db67 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:59.592810 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cffef3-3e8d-4cfb-bbd9-33d7bd50db67 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:59.632069 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=6,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:03:59.716319 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd944a10-7cc6-4cb2-9a62-af53750977e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:59.717530 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd944a10-7cc6-4cb2-9a62-af53750977e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:59.717530 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd944a10-7cc6-4cb2-9a62-af53750977e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:59.717868 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd944a10-7cc6-4cb2-9a62-af53750977e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:59.718583 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd944a10-7cc6-4cb2-9a62-af53750977e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:59.720998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd944a10-7cc6-4cb2-9a62-af53750977e5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:59.738680 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1091/2222] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:03:59 2022] GET /identity/v3/roles => generated 1420 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 01:03:59.739532 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b55f107-edd3-4eb2-8cf5-e4e52b561f5c admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:59.751362 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b55f107-edd3-4eb2-8cf5-e4e52b561f5c admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:03:59.768862 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8608e4c-ff30-403d-a7a4-b03682c9d8bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:03:59.769560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8608e4c-ff30-403d-a7a4-b03682c9d8bf admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:59.769844 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8608e4c-ff30-403d-a7a4-b03682c9d8bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:59.770122 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8608e4c-ff30-403d-a7a4-b03682c9d8bf admin admin] PATH_INFO: `/v3/projects/68b1319516144f958fd2daf6fc42b42e/users/9eec3b89e5934cad930bc5fb89aa91c8/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:59.773545 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8608e4c-ff30-403d-a7a4-b03682c9d8bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=68b1319516144f958fd2daf6fc42b42e, user_id=9eec3b89e5934cad930bc5fb89aa91c8, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:03:59.776422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8608e4c-ff30-403d-a7a4-b03682c9d8bf admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:03:59.795690 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1132/2223] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:03:59 2022] PUT /identity/v3/projects/68b1319516144f958fd2daf6fc42b42e/users/9eec3b89e5934cad930bc5fb89aa91c8/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:03:59.807648 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743ecf3d-f5dd-4581-97ec-808780effea3 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:03:59.807937 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743ecf3d-f5dd-4581-97ec-808780effea3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:03:59.808212 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743ecf3d-f5dd-4581-97ec-808780effea3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:03:59.848791 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-743ecf3d-f5dd-4581-97ec-808780effea3 None None] MFA Rules not processed for user `9eec3b89e5934cad930bc5fb89aa91c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:03:59.850122 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-743ecf3d-f5dd-4581-97ec-808780effea3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:59.946231 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1092/2224] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:03:59 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:03:59.959382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8608e4c-ff30-403d-a7a4-b03682c9d8bf admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:03:59.961373 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d8608e4c-ff30-403d-a7a4-b03682c9d8bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:03:59.992174 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:INSERT=8,keystone:SELECT=92 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:00.036653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8608e4c-ff30-403d-a7a4-b03682c9d8bf admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:00.040648 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03e8724-6337-4932-8baf-4329f6a433df service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:00.041271 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03e8724-6337-4932-8baf-4329f6a433df service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:00.041587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03e8724-6337-4932-8baf-4329f6a433df service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:00.041963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03e8724-6337-4932-8baf-4329f6a433df service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:00.056797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03e8724-6337-4932-8baf-4329f6a433df service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:00.058393 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03e8724-6337-4932-8baf-4329f6a433df service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:00.068969 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1133/2225] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:03:59 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:00.150405 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743ecf3d-f5dd-4581-97ec-808780effea3 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:00.160179 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743ecf3d-f5dd-4581-97ec-808780effea3 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:00.164529 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9617749-c98b-441d-be31-bffd85bed393 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:00.165150 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9617749-c98b-441d-be31-bffd85bed393 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:00.165432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9617749-c98b-441d-be31-bffd85bed393 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:00.165705 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9617749-c98b-441d-be31-bffd85bed393 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:00.231335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9617749-c98b-441d-be31-bffd85bed393 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:00.232830 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9617749-c98b-441d-be31-bffd85bed393 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:00.243188 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1093/2226] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:00 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:03.858803 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03e8724-6337-4932-8baf-4329f6a433df service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:03.860380 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b03e8724-6337-4932-8baf-4329f6a433df service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:03.926740 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03e8724-6337-4932-8baf-4329f6a433df service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:03.931906 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4685267-c92e-4fbc-b727-674ca51d9ffc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:03.932646 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4685267-c92e-4fbc-b727-674ca51d9ffc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:03.932828 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4685267-c92e-4fbc-b727-674ca51d9ffc admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:03.933087 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4685267-c92e-4fbc-b727-674ca51d9ffc admin admin] PATH_INFO: `/v3/users/a49486f43ff942bcbf4ef782d6eb6238` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:03.934191 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4685267-c92e-4fbc-b727-674ca51d9ffc admin admin] RBAC: Authorizing `identity:delete_user(user_id=a49486f43ff942bcbf4ef782d6eb6238)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:03.936152 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4685267-c92e-4fbc-b727-674ca51d9ffc admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:03.982992 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c4685267-c92e-4fbc-b727-674ca51d9ffc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a49486f43ff942bcbf4ef782d6eb6238'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:03.983681 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c4685267-c92e-4fbc-b727-674ca51d9ffc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a49486f43ff942bcbf4ef782d6eb6238'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:03.991625 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c4685267-c92e-4fbc-b727-674ca51d9ffc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a49486f43ff942bcbf4ef782d6eb6238'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.004262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c4685267-c92e-4fbc-b727-674ca51d9ffc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a49486f43ff942bcbf4ef782d6eb6238'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.011974 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c4685267-c92e-4fbc-b727-674ca51d9ffc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a49486f43ff942bcbf4ef782d6eb6238'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.017965 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1134/2227] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:04:03 2022] DELETE /identity/v3/users/a49486f43ff942bcbf4ef782d6eb6238 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:04:04.186816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9617749-c98b-441d-be31-bffd85bed393 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:04.198840 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9617749-c98b-441d-be31-bffd85bed393 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:04.243545 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:04.260977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:04.261617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:04.261925 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:04.262203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a admin admin] PATH_INFO: `/v3/projects/343c019304314c669a3381cc4c68e481` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:04.263362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a admin admin] RBAC: Authorizing `identity:delete_project(project_id=343c019304314c669a3381cc4c68e481)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:04.265304 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:04.311591 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '343c019304314c669a3381cc4c68e481', 'request_id': 'req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.319762 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '343c019304314c669a3381cc4c68e481', 'request_id': 'req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.326701 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '343c019304314c669a3381cc4c68e481', 'request_id': 'req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.337725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a admin admin] The token cache is being invalidate because project 343c019304314c669a3381cc4c68e481 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:04:04.338013 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.338660 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.340192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1094/2228] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:04:04 2022] DELETE /identity/v3/projects/343c019304314c669a3381cc4c68e481 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:04.344341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4685267-c92e-4fbc-b727-674ca51d9ffc admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:04.345756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c4685267-c92e-4fbc-b727-674ca51d9ffc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:04.359078 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4685267-c92e-4fbc-b727-674ca51d9ffc admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:04.400566 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:04.421162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096710d0-ec43-4f57-a5e9-854a522af7c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:04.421965 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096710d0-ec43-4f57-a5e9-854a522af7c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:04.422090 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096710d0-ec43-4f57-a5e9-854a522af7c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:04.422362 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096710d0-ec43-4f57-a5e9-854a522af7c9 admin admin] PATH_INFO: `/v3/users/9a16b4d57e56464e91a8cb5b1fb8950c` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:04.423551 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096710d0-ec43-4f57-a5e9-854a522af7c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9a16b4d57e56464e91a8cb5b1fb8950c)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:04.425519 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096710d0-ec43-4f57-a5e9-854a522af7c9 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:04.477611 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-096710d0-ec43-4f57-a5e9-854a522af7c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9a16b4d57e56464e91a8cb5b1fb8950c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.478294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-096710d0-ec43-4f57-a5e9-854a522af7c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9a16b4d57e56464e91a8cb5b1fb8950c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.489408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-096710d0-ec43-4f57-a5e9-854a522af7c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9a16b4d57e56464e91a8cb5b1fb8950c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.504918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-096710d0-ec43-4f57-a5e9-854a522af7c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9a16b4d57e56464e91a8cb5b1fb8950c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.513008 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-096710d0-ec43-4f57-a5e9-854a522af7c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9a16b4d57e56464e91a8cb5b1fb8950c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.519012 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1135/2229] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:04:04 2022] DELETE /identity/v3/users/9a16b4d57e56464e91a8cb5b1fb8950c => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:04.581598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:04.583059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:04.601524 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286e9924-4e7f-4180-b1c8-84c9d7d3d23a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:04.677857 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbdf37e3-bffd-4e67-88e8-a16e7627c8bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:04.678550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdf37e3-bffd-4e67-88e8-a16e7627c8bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:04.679002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdf37e3-bffd-4e67-88e8-a16e7627c8bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:04.679281 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbdf37e3-bffd-4e67-88e8-a16e7627c8bf admin admin] PATH_INFO: `/v3/users/25165bf99f82450eabf4a91bbd3b8641` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:04.680468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbdf37e3-bffd-4e67-88e8-a16e7627c8bf admin admin] RBAC: Authorizing `identity:delete_user(user_id=25165bf99f82450eabf4a91bbd3b8641)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:04.682643 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbdf37e3-bffd-4e67-88e8-a16e7627c8bf admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:04.748602 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096710d0-ec43-4f57-a5e9-854a522af7c9 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:04.750588 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-096710d0-ec43-4f57-a5e9-854a522af7c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:04.775338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096710d0-ec43-4f57-a5e9-854a522af7c9 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:04.779460 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-dbdf37e3-bffd-4e67-88e8-a16e7627c8bf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '25165bf99f82450eabf4a91bbd3b8641'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.780164 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-dbdf37e3-bffd-4e67-88e8-a16e7627c8bf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '25165bf99f82450eabf4a91bbd3b8641'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.786172 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735191f7-ee19-4118-8afd-ce6abe7cadc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:04.791453 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-dbdf37e3-bffd-4e67-88e8-a16e7627c8bf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '25165bf99f82450eabf4a91bbd3b8641'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.792407 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-dbdf37e3-bffd-4e67-88e8-a16e7627c8bf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '25165bf99f82450eabf4a91bbd3b8641'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.797621 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735191f7-ee19-4118-8afd-ce6abe7cadc2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:04.798524 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735191f7-ee19-4118-8afd-ce6abe7cadc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:04.798889 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735191f7-ee19-4118-8afd-ce6abe7cadc2 admin admin] PATH_INFO: `/v3/projects/07fd28660dad4acf93dc9785e33d2366` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:04.801027 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735191f7-ee19-4118-8afd-ce6abe7cadc2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=07fd28660dad4acf93dc9785e33d2366)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:04.803811 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735191f7-ee19-4118-8afd-ce6abe7cadc2 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:04.845680 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-dbdf37e3-bffd-4e67-88e8-a16e7627c8bf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '25165bf99f82450eabf4a91bbd3b8641'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.847881 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:04.862965 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1095/2230] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:04:04 2022] DELETE /identity/v3/users/25165bf99f82450eabf4a91bbd3b8641 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:04.868417 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213a1eee-c561-418b-864e-b45300bfdea7 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:04.868712 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213a1eee-c561-418b-864e-b45300bfdea7 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:04.868988 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213a1eee-c561-418b-864e-b45300bfdea7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:04.878590 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-735191f7-ee19-4118-8afd-ce6abe7cadc2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '07fd28660dad4acf93dc9785e33d2366', 'request_id': 'req-735191f7-ee19-4118-8afd-ce6abe7cadc2'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.888831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-735191f7-ee19-4118-8afd-ce6abe7cadc2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '07fd28660dad4acf93dc9785e33d2366', 'request_id': 'req-735191f7-ee19-4118-8afd-ce6abe7cadc2'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.889681 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-213a1eee-c561-418b-864e-b45300bfdea7 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:04.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-213a1eee-c561-418b-864e-b45300bfdea7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:04.897760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-735191f7-ee19-4118-8afd-ce6abe7cadc2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '07fd28660dad4acf93dc9785e33d2366', 'request_id': 'req-735191f7-ee19-4118-8afd-ce6abe7cadc2'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.912234 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-735191f7-ee19-4118-8afd-ce6abe7cadc2 admin admin] The token cache is being invalidate because project 07fd28660dad4acf93dc9785e33d2366 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:04:04.912565 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-735191f7-ee19-4118-8afd-ce6abe7cadc2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.913294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-735191f7-ee19-4118-8afd-ce6abe7cadc2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:04.927390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1136/2231] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:04:04 2022] DELETE /identity/v3/projects/07fd28660dad4acf93dc9785e33d2366 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:04:04.966113 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1096/2232] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:04.979435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735191f7-ee19-4118-8afd-ce6abe7cadc2 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:04.981102 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-735191f7-ee19-4118-8afd-ce6abe7cadc2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:05.041454 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:05.058062 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735191f7-ee19-4118-8afd-ce6abe7cadc2 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:05.063947 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021053bd-afc7-40ae-a9df-fa3d1fa0902e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:05.064661 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021053bd-afc7-40ae-a9df-fa3d1fa0902e service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:05.065170 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021053bd-afc7-40ae-a9df-fa3d1fa0902e service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:05.065520 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021053bd-afc7-40ae-a9df-fa3d1fa0902e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:05.076808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021053bd-afc7-40ae-a9df-fa3d1fa0902e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:05.078589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021053bd-afc7-40ae-a9df-fa3d1fa0902e service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:05.090791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1137/2233] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:05.375402 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213a1eee-c561-418b-864e-b45300bfdea7 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:05.376809 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-213a1eee-c561-418b-864e-b45300bfdea7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:05.390123 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213a1eee-c561-418b-864e-b45300bfdea7 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:05.394827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6244c374-2a47-4aa1-811f-54cf478da4a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:05.395507 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6244c374-2a47-4aa1-811f-54cf478da4a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:05.395790 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6244c374-2a47-4aa1-811f-54cf478da4a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:05.396072 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6244c374-2a47-4aa1-811f-54cf478da4a0 admin admin] PATH_INFO: `/v3/projects/5fe7780fa7f141e89f06bc9e9cbb7b78` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:05.397900 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6244c374-2a47-4aa1-811f-54cf478da4a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5fe7780fa7f141e89f06bc9e9cbb7b78)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:05.400243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6244c374-2a47-4aa1-811f-54cf478da4a0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:05.450183 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6244c374-2a47-4aa1-811f-54cf478da4a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5fe7780fa7f141e89f06bc9e9cbb7b78', 'request_id': 'req-6244c374-2a47-4aa1-811f-54cf478da4a0'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:05.456797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6244c374-2a47-4aa1-811f-54cf478da4a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5fe7780fa7f141e89f06bc9e9cbb7b78', 'request_id': 'req-6244c374-2a47-4aa1-811f-54cf478da4a0'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:05.464300 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6244c374-2a47-4aa1-811f-54cf478da4a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5fe7780fa7f141e89f06bc9e9cbb7b78', 'request_id': 'req-6244c374-2a47-4aa1-811f-54cf478da4a0'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:05.474709 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6244c374-2a47-4aa1-811f-54cf478da4a0 admin admin] The token cache is being invalidate because project 5fe7780fa7f141e89f06bc9e9cbb7b78 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:04:05.475680 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6244c374-2a47-4aa1-811f-54cf478da4a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:05.476581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6244c374-2a47-4aa1-811f-54cf478da4a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:05.478072 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1097/2234] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:04:05 2022] DELETE /identity/v3/projects/5fe7780fa7f141e89f06bc9e9cbb7b78 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:04:05.482136 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021053bd-afc7-40ae-a9df-fa3d1fa0902e service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:05.492150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021053bd-afc7-40ae-a9df-fa3d1fa0902e service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:05.549666 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebd1db7-ed39-4303-8253-fecd4ca1a118 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:05.549968 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebd1db7-ed39-4303-8253-fecd4ca1a118 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:05.550252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebd1db7-ed39-4303-8253-fecd4ca1a118 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:05.550532 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebd1db7-ed39-4303-8253-fecd4ca1a118 admin admin] PATH_INFO: `/v3/users/9eec3b89e5934cad930bc5fb89aa91c8` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:05.551798 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebd1db7-ed39-4303-8253-fecd4ca1a118 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9eec3b89e5934cad930bc5fb89aa91c8)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:05.554075 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebd1db7-ed39-4303-8253-fecd4ca1a118 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:05.602700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-5ebd1db7-ed39-4303-8253-fecd4ca1a118 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9eec3b89e5934cad930bc5fb89aa91c8'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:05.603778 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-5ebd1db7-ed39-4303-8253-fecd4ca1a118 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9eec3b89e5934cad930bc5fb89aa91c8'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:05.613249 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-5ebd1db7-ed39-4303-8253-fecd4ca1a118 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9eec3b89e5934cad930bc5fb89aa91c8'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:05.625692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-5ebd1db7-ed39-4303-8253-fecd4ca1a118 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9eec3b89e5934cad930bc5fb89aa91c8'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:05.632909 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-5ebd1db7-ed39-4303-8253-fecd4ca1a118 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9eec3b89e5934cad930bc5fb89aa91c8'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:05.638827 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1138/2235] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:04:05 2022] DELETE /identity/v3/users/9eec3b89e5934cad930bc5fb89aa91c8 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:05.922647 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6244c374-2a47-4aa1-811f-54cf478da4a0 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:05.924136 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6244c374-2a47-4aa1-811f-54cf478da4a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:05.943377 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6244c374-2a47-4aa1-811f-54cf478da4a0 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:06.005876 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3f6a0d-131b-4b1c-8787-353c62963bec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:06.007129 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3f6a0d-131b-4b1c-8787-353c62963bec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:06.007434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3f6a0d-131b-4b1c-8787-353c62963bec admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:06.007666 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3f6a0d-131b-4b1c-8787-353c62963bec admin admin] PATH_INFO: `/v3/projects/68b1319516144f958fd2daf6fc42b42e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:06.008794 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3f6a0d-131b-4b1c-8787-353c62963bec admin admin] RBAC: Authorizing `identity:delete_project(project_id=68b1319516144f958fd2daf6fc42b42e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:06.010706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3f6a0d-131b-4b1c-8787-353c62963bec admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:06.028607 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:06.059860 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1a3f6a0d-131b-4b1c-8787-353c62963bec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '68b1319516144f958fd2daf6fc42b42e', 'request_id': 'req-1a3f6a0d-131b-4b1c-8787-353c62963bec'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:06.067121 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1a3f6a0d-131b-4b1c-8787-353c62963bec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '68b1319516144f958fd2daf6fc42b42e', 'request_id': 'req-1a3f6a0d-131b-4b1c-8787-353c62963bec'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:06.074712 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1a3f6a0d-131b-4b1c-8787-353c62963bec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '68b1319516144f958fd2daf6fc42b42e', 'request_id': 'req-1a3f6a0d-131b-4b1c-8787-353c62963bec'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:06.085547 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1a3f6a0d-131b-4b1c-8787-353c62963bec admin admin] The token cache is being invalidate because project 68b1319516144f958fd2daf6fc42b42e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:04:06.085833 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1a3f6a0d-131b-4b1c-8787-353c62963bec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:06.086481 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-1a3f6a0d-131b-4b1c-8787-353c62963bec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:06.087986 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1098/2236] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:04:05 2022] DELETE /identity/v3/projects/68b1319516144f958fd2daf6fc42b42e => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:06.093144 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06773b38-1f35-4c1e-88b5-84cf0af8e975 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:06.093619 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06773b38-1f35-4c1e-88b5-84cf0af8e975 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:06.093736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06773b38-1f35-4c1e-88b5-84cf0af8e975 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:06.113729 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-06773b38-1f35-4c1e-88b5-84cf0af8e975 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:06.114991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-06773b38-1f35-4c1e-88b5-84cf0af8e975 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:06.175302 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1139/2237] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:06.184009 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484a27da-2b07-4218-abc2-babfdf3e3aca None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:06.184353 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484a27da-2b07-4218-abc2-babfdf3e3aca None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:06.184564 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484a27da-2b07-4218-abc2-babfdf3e3aca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:06.207617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-484a27da-2b07-4218-abc2-babfdf3e3aca None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:06.208604 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-484a27da-2b07-4218-abc2-babfdf3e3aca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:06.216826 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1099/2238] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:06.221229 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06773b38-1f35-4c1e-88b5-84cf0af8e975 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:06.231452 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06773b38-1f35-4c1e-88b5-84cf0af8e975 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:06.236052 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50fc017-8b0f-42f0-adcc-a00dff85014d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:06.236677 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50fc017-8b0f-42f0-adcc-a00dff85014d admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:06.236958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50fc017-8b0f-42f0-adcc-a00dff85014d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:06.237230 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50fc017-8b0f-42f0-adcc-a00dff85014d admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:06.237873 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50fc017-8b0f-42f0-adcc-a00dff85014d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:04:06.238205 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50fc017-8b0f-42f0-adcc-a00dff85014d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:06.239847 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50fc017-8b0f-42f0-adcc-a00dff85014d admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:06.242251 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:06.254559 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1140/2239] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:04:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:04:06.260083 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484a27da-2b07-4218-abc2-babfdf3e3aca None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:06.269384 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484a27da-2b07-4218-abc2-babfdf3e3aca None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:06.274125 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91562cea-34bc-4859-9715-24f12b6baaa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:06.274723 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91562cea-34bc-4859-9715-24f12b6baaa5 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:06.275047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91562cea-34bc-4859-9715-24f12b6baaa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:06.275321 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91562cea-34bc-4859-9715-24f12b6baaa5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:06.275958 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91562cea-34bc-4859-9715-24f12b6baaa5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:06.277875 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91562cea-34bc-4859-9715-24f12b6baaa5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:06.305435 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1100/2240] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:04:06 2022] POST /identity/v3/projects => generated 385 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:04:06.309736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50fc017-8b0f-42f0-adcc-a00dff85014d admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:06.320433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50fc017-8b0f-42f0-adcc-a00dff85014d admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:06.377783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1dbebf-9944-4a30-9e5c-6e7d781a499e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:06.378309 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1dbebf-9944-4a30-9e5c-6e7d781a499e admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:06.378587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1dbebf-9944-4a30-9e5c-6e7d781a499e admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:06.378869 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1dbebf-9944-4a30-9e5c-6e7d781a499e admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:06.379511 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1dbebf-9944-4a30-9e5c-6e7d781a499e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:06.382072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1dbebf-9944-4a30-9e5c-6e7d781a499e admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:06.402190 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1141/2241] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:04:06 2022] POST /identity/v3/users => generated 293 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:04:06.406837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91562cea-34bc-4859-9715-24f12b6baaa5 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:06.417576 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91562cea-34bc-4859-9715-24f12b6baaa5 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:06.422079 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93a1fd6-6ed5-4cd8-a2a6-1e31da21fe23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:06.422675 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93a1fd6-6ed5-4cd8-a2a6-1e31da21fe23 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:06.422987 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93a1fd6-6ed5-4cd8-a2a6-1e31da21fe23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:06.423262 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93a1fd6-6ed5-4cd8-a2a6-1e31da21fe23 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:06.423903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93a1fd6-6ed5-4cd8-a2a6-1e31da21fe23 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:06.425515 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93a1fd6-6ed5-4cd8-a2a6-1e31da21fe23 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:06.436657 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1101/2242] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:06 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:04:06.440660 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1dbebf-9944-4a30-9e5c-6e7d781a499e admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:06.451420 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1dbebf-9944-4a30-9e5c-6e7d781a499e admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:06.455941 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3822ec17-ecf8-4240-9779-8dde985e6753 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:06.456534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3822ec17-ecf8-4240-9779-8dde985e6753 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:06.456810 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3822ec17-ecf8-4240-9779-8dde985e6753 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:06.457079 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3822ec17-ecf8-4240-9779-8dde985e6753 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:06.457764 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3822ec17-ecf8-4240-9779-8dde985e6753 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:06.459481 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3822ec17-ecf8-4240-9779-8dde985e6753 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:06.470296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1142/2243] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:06 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:04:06.474596 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93a1fd6-6ed5-4cd8-a2a6-1e31da21fe23 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:06.485867 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93a1fd6-6ed5-4cd8-a2a6-1e31da21fe23 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:06.490536 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb984bcf-6d57-416f-b824-20ddbd29c00d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:06.491943 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb984bcf-6d57-416f-b824-20ddbd29c00d admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:06.492209 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb984bcf-6d57-416f-b824-20ddbd29c00d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:06.492510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb984bcf-6d57-416f-b824-20ddbd29c00d admin admin] PATH_INFO: `/v3/projects/fc46802e590646148357aec816608ed7/users/eca314685ddf40fdaf09fadc6e0c7586/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:06.507181 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb984bcf-6d57-416f-b824-20ddbd29c00d admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc46802e590646148357aec816608ed7, user_id=eca314685ddf40fdaf09fadc6e0c7586, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:06.509865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb984bcf-6d57-416f-b824-20ddbd29c00d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:06.523683 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1102/2244] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:04:06 2022] PUT /identity/v3/projects/fc46802e590646148357aec816608ed7/users/eca314685ddf40fdaf09fadc6e0c7586/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:04:06.534935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80409109-4aa1-431c-97f3-eb1635161f1b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:06.535229 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80409109-4aa1-431c-97f3-eb1635161f1b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:06.535509 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80409109-4aa1-431c-97f3-eb1635161f1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:06.567738 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-80409109-4aa1-431c-97f3-eb1635161f1b None None] MFA Rules not processed for user `eca314685ddf40fdaf09fadc6e0c7586`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:06.569193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-80409109-4aa1-431c-97f3-eb1635161f1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:06.640262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1143/2245] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:06 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:06.651832 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb984bcf-6d57-416f-b824-20ddbd29c00d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:06.653763 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-fb984bcf-6d57-416f-b824-20ddbd29c00d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:06.711224 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb984bcf-6d57-416f-b824-20ddbd29c00d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:06.714837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7675d444-28fd-483c-8546-ec00578a3cc0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:06.715459 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7675d444-28fd-483c-8546-ec00578a3cc0 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:06.715734 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7675d444-28fd-483c-8546-ec00578a3cc0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:06.716002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7675d444-28fd-483c-8546-ec00578a3cc0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:06.724601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7675d444-28fd-483c-8546-ec00578a3cc0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:06.726739 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7675d444-28fd-483c-8546-ec00578a3cc0 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:06.735598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1103/2246] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:06 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:06.757122 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80409109-4aa1-431c-97f3-eb1635161f1b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:06.818414 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80409109-4aa1-431c-97f3-eb1635161f1b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:06.823391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a34e81-41b9-4aa6-a68f-554fc3bf122d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:06.824016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a34e81-41b9-4aa6-a68f-554fc3bf122d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:06.824299 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a34e81-41b9-4aa6-a68f-554fc3bf122d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:06.824574 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a34e81-41b9-4aa6-a68f-554fc3bf122d admin admin] PATH_INFO: `/v3/users/eca314685ddf40fdaf09fadc6e0c7586` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:06.825699 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a34e81-41b9-4aa6-a68f-554fc3bf122d admin admin] RBAC: Authorizing `identity:delete_user(user_id=eca314685ddf40fdaf09fadc6e0c7586)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:06.827669 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a34e81-41b9-4aa6-a68f-554fc3bf122d admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:06.830009 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:06.878248 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-50a34e81-41b9-4aa6-a68f-554fc3bf122d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eca314685ddf40fdaf09fadc6e0c7586'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:06.878964 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-50a34e81-41b9-4aa6-a68f-554fc3bf122d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eca314685ddf40fdaf09fadc6e0c7586'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:06.887027 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-50a34e81-41b9-4aa6-a68f-554fc3bf122d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eca314685ddf40fdaf09fadc6e0c7586'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:06.899673 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-50a34e81-41b9-4aa6-a68f-554fc3bf122d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eca314685ddf40fdaf09fadc6e0c7586'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:06.906453 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-50a34e81-41b9-4aa6-a68f-554fc3bf122d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eca314685ddf40fdaf09fadc6e0c7586'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:06.912371 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1144/2247] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:04:06 2022] DELETE /identity/v3/users/eca314685ddf40fdaf09fadc6e0c7586 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:06.917363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbd995e-21cf-4e33-9405-17b385686057 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:06.917650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbd995e-21cf-4e33-9405-17b385686057 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:06.917926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbd995e-21cf-4e33-9405-17b385686057 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:06.936755 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-bbbd995e-21cf-4e33-9405-17b385686057 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:06.937943 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bbbd995e-21cf-4e33-9405-17b385686057 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:06.979122 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:07.000966 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1104/2248] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:07.010949 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a34e81-41b9-4aa6-a68f-554fc3bf122d admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:07.012373 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-50a34e81-41b9-4aa6-a68f-554fc3bf122d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:07.070777 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a34e81-41b9-4aa6-a68f-554fc3bf122d admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:07.074704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9a36c2-6aad-4d1b-9b99-399174a70e18 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:07.075347 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9a36c2-6aad-4d1b-9b99-399174a70e18 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:07.075625 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9a36c2-6aad-4d1b-9b99-399174a70e18 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:07.075897 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9a36c2-6aad-4d1b-9b99-399174a70e18 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:07.097554 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9a36c2-6aad-4d1b-9b99-399174a70e18 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:07.099190 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9a36c2-6aad-4d1b-9b99-399174a70e18 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:07.109862 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1145/2249] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:07.449524 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbd995e-21cf-4e33-9405-17b385686057 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:07.451496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-bbbd995e-21cf-4e33-9405-17b385686057 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:07.466202 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbd995e-21cf-4e33-9405-17b385686057 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:07.470847 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2462b968-8083-4c0b-91ba-f5cb0492aed2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:07.471557 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2462b968-8083-4c0b-91ba-f5cb0492aed2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:07.471845 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2462b968-8083-4c0b-91ba-f5cb0492aed2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:07.472120 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2462b968-8083-4c0b-91ba-f5cb0492aed2 admin admin] PATH_INFO: `/v3/projects/fc46802e590646148357aec816608ed7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:07.473258 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2462b968-8083-4c0b-91ba-f5cb0492aed2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc46802e590646148357aec816608ed7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:07.475244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2462b968-8083-4c0b-91ba-f5cb0492aed2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:07.524053 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2462b968-8083-4c0b-91ba-f5cb0492aed2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc46802e590646148357aec816608ed7', 'request_id': 'req-2462b968-8083-4c0b-91ba-f5cb0492aed2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:07.531780 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2462b968-8083-4c0b-91ba-f5cb0492aed2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc46802e590646148357aec816608ed7', 'request_id': 'req-2462b968-8083-4c0b-91ba-f5cb0492aed2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:07.538649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2462b968-8083-4c0b-91ba-f5cb0492aed2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc46802e590646148357aec816608ed7', 'request_id': 'req-2462b968-8083-4c0b-91ba-f5cb0492aed2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:07.549158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2462b968-8083-4c0b-91ba-f5cb0492aed2 admin admin] The token cache is being invalidate because project fc46802e590646148357aec816608ed7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:04:07.549445 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2462b968-8083-4c0b-91ba-f5cb0492aed2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:07.550086 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-2462b968-8083-4c0b-91ba-f5cb0492aed2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:07.551615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1105/2250] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:04:07 2022] DELETE /identity/v3/projects/fc46802e590646148357aec816608ed7 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:07.557369 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4312ecb4-5903-4a8d-8f8b-baf8b8b5b3fa None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:07.557647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4312ecb4-5903-4a8d-8f8b-baf8b8b5b3fa None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:07.557931 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4312ecb4-5903-4a8d-8f8b-baf8b8b5b3fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:07.578268 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-4312ecb4-5903-4a8d-8f8b-baf8b8b5b3fa None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:07.579518 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4312ecb4-5903-4a8d-8f8b-baf8b8b5b3fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:07.643764 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1146/2251] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:07.651531 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efedbb0-294d-48ba-a43d-109b9e8ee6da None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:07.651816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efedbb0-294d-48ba-a43d-109b9e8ee6da None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:07.652117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efedbb0-294d-48ba-a43d-109b9e8ee6da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:07.671716 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-0efedbb0-294d-48ba-a43d-109b9e8ee6da None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:07.672906 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0efedbb0-294d-48ba-a43d-109b9e8ee6da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:07.681529 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1106/2252] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:07.685737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4312ecb4-5903-4a8d-8f8b-baf8b8b5b3fa None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:07.695558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4312ecb4-5903-4a8d-8f8b-baf8b8b5b3fa None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:07.701077 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fa64aa-9186-42d6-9f7f-3aa12ef08091 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:07.701682 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fa64aa-9186-42d6-9f7f-3aa12ef08091 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:07.701958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fa64aa-9186-42d6-9f7f-3aa12ef08091 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:07.702229 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fa64aa-9186-42d6-9f7f-3aa12ef08091 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:07.702887 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fa64aa-9186-42d6-9f7f-3aa12ef08091 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:04:07.703202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fa64aa-9186-42d6-9f7f-3aa12ef08091 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:07.704793 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fa64aa-9186-42d6-9f7f-3aa12ef08091 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:07.716574 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1147/2253] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:04:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 01:04:07.721112 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0efedbb0-294d-48ba-a43d-109b9e8ee6da None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:07.731345 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0efedbb0-294d-48ba-a43d-109b9e8ee6da None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:07.736070 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076ab363-071c-4a67-8aa9-a6c30d0cc538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:07.736668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076ab363-071c-4a67-8aa9-a6c30d0cc538 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:07.736945 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076ab363-071c-4a67-8aa9-a6c30d0cc538 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:07.737216 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076ab363-071c-4a67-8aa9-a6c30d0cc538 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:07.737851 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076ab363-071c-4a67-8aa9-a6c30d0cc538 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:07.739827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076ab363-071c-4a67-8aa9-a6c30d0cc538 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:07.767928 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1107/2254] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:04:07 2022] POST /identity/v3/projects => generated 377 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 01:04:07.772605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fa64aa-9186-42d6-9f7f-3aa12ef08091 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:07.782437 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fa64aa-9186-42d6-9f7f-3aa12ef08091 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:07.790211 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:07.841067 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a74a6de-4070-4189-96e8-a59d9151d147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:07.841686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a74a6de-4070-4189-96e8-a59d9151d147 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:07.841974 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a74a6de-4070-4189-96e8-a59d9151d147 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:07.842251 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a74a6de-4070-4189-96e8-a59d9151d147 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:07.842973 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a74a6de-4070-4189-96e8-a59d9151d147 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:07.845025 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a74a6de-4070-4189-96e8-a59d9151d147 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:07.866813 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1148/2255] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:04:07 2022] POST /identity/v3/users => generated 289 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:04:07.870933 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076ab363-071c-4a67-8aa9-a6c30d0cc538 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:07.881091 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076ab363-071c-4a67-8aa9-a6c30d0cc538 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:07.886470 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00353b10-789c-447f-bae5-c95b58d2c0d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:07.887110 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00353b10-789c-447f-bae5-c95b58d2c0d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:07.887498 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00353b10-789c-447f-bae5-c95b58d2c0d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:07.887776 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00353b10-789c-447f-bae5-c95b58d2c0d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:07.888418 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00353b10-789c-447f-bae5-c95b58d2c0d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:07.890023 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00353b10-789c-447f-bae5-c95b58d2c0d4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:07.900027 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1108/2256] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:04:07.903931 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a74a6de-4070-4189-96e8-a59d9151d147 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:07.914149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a74a6de-4070-4189-96e8-a59d9151d147 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:07.918847 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e6aeab-cac9-4d1f-ac2f-5bb41a436bf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:07.919515 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e6aeab-cac9-4d1f-ac2f-5bb41a436bf2 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:07.919795 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e6aeab-cac9-4d1f-ac2f-5bb41a436bf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:07.920071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e6aeab-cac9-4d1f-ac2f-5bb41a436bf2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:07.920717 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e6aeab-cac9-4d1f-ac2f-5bb41a436bf2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:07.922451 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e6aeab-cac9-4d1f-ac2f-5bb41a436bf2 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:07.934018 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1149/2257] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:04:07.938103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00353b10-789c-447f-bae5-c95b58d2c0d4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:07.949023 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00353b10-789c-447f-bae5-c95b58d2c0d4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:07.954650 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1f40f9-d70d-49d3-8b30-e0c463cd2298 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:07.955343 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1f40f9-d70d-49d3-8b30-e0c463cd2298 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:07.955616 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1f40f9-d70d-49d3-8b30-e0c463cd2298 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:07.955893 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa1f40f9-d70d-49d3-8b30-e0c463cd2298 admin admin] PATH_INFO: `/v3/projects/f9ad83b53a08431696ba4fd2f6d519eb/users/ab09167b28a147ce8fed799221b91674/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:07.973087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1f40f9-d70d-49d3-8b30-e0c463cd2298 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f9ad83b53a08431696ba4fd2f6d519eb, user_id=ab09167b28a147ce8fed799221b91674, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:07.977243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa1f40f9-d70d-49d3-8b30-e0c463cd2298 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:07.991283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1109/2258] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:04:07 2022] PUT /identity/v3/projects/f9ad83b53a08431696ba4fd2f6d519eb/users/ab09167b28a147ce8fed799221b91674/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:08.001526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5d4f0c-8b67-4018-bf78-c670deb9dc07 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:08.001816 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5d4f0c-8b67-4018-bf78-c670deb9dc07 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:08.002092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5d4f0c-8b67-4018-bf78-c670deb9dc07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:08.029720 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-fe5d4f0c-8b67-4018-bf78-c670deb9dc07 None None] MFA Rules not processed for user `ab09167b28a147ce8fed799221b91674`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:08.030977 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-fe5d4f0c-8b67-4018-bf78-c670deb9dc07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:08.095628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1150/2259] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:08 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:08.108013 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1f40f9-d70d-49d3-8b30-e0c463cd2298 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:08.109722 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-aa1f40f9-d70d-49d3-8b30-e0c463cd2298 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:08.167343 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa1f40f9-d70d-49d3-8b30-e0c463cd2298 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:08.171081 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fddb52c-489b-45ef-84d9-5651a98a19fe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:08.171697 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fddb52c-489b-45ef-84d9-5651a98a19fe service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:08.171979 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fddb52c-489b-45ef-84d9-5651a98a19fe service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:08.172256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fddb52c-489b-45ef-84d9-5651a98a19fe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:08.181371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fddb52c-489b-45ef-84d9-5651a98a19fe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:08.182819 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fddb52c-489b-45ef-84d9-5651a98a19fe service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:08.192622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1110/2260] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:08 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:18.092714 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=61,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:18.188092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=87,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:26.872357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5d4f0c-8b67-4018-bf78-c670deb9dc07 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:26.934695 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5d4f0c-8b67-4018-bf78-c670deb9dc07 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:26.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607d054-aac8-4e31-a7d9-6507d03a78b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:26.939895 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607d054-aac8-4e31-a7d9-6507d03a78b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:26.940180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607d054-aac8-4e31-a7d9-6507d03a78b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:26.940460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607d054-aac8-4e31-a7d9-6507d03a78b3 admin admin] PATH_INFO: `/v3/users/ab09167b28a147ce8fed799221b91674` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:26.941592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e607d054-aac8-4e31-a7d9-6507d03a78b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ab09167b28a147ce8fed799221b91674)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:26.943610 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e607d054-aac8-4e31-a7d9-6507d03a78b3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:26.992690 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e607d054-aac8-4e31-a7d9-6507d03a78b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ab09167b28a147ce8fed799221b91674'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:26.993347 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e607d054-aac8-4e31-a7d9-6507d03a78b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ab09167b28a147ce8fed799221b91674'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:27.000081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e607d054-aac8-4e31-a7d9-6507d03a78b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ab09167b28a147ce8fed799221b91674'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:27.012481 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e607d054-aac8-4e31-a7d9-6507d03a78b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ab09167b28a147ce8fed799221b91674'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:27.019411 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-e607d054-aac8-4e31-a7d9-6507d03a78b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ab09167b28a147ce8fed799221b91674'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:27.025634 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1151/2261] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:04:26 2022] DELETE /identity/v3/users/ab09167b28a147ce8fed799221b91674 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:27.030108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3571cfc3-bd5f-44f1-bac0-df644fbcba88 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:27.030610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3571cfc3-bd5f-44f1-bac0-df644fbcba88 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:27.030721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3571cfc3-bd5f-44f1-bac0-df644fbcba88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:27.050169 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3571cfc3-bd5f-44f1-bac0-df644fbcba88 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:27.051406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3571cfc3-bd5f-44f1-bac0-df644fbcba88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:27.113197 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1111/2262] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:27.123229 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607d054-aac8-4e31-a7d9-6507d03a78b3 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:27.124646 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-e607d054-aac8-4e31-a7d9-6507d03a78b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:27.182504 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607d054-aac8-4e31-a7d9-6507d03a78b3 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:27.185149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718bbfe3-7176-4a09-b380-19b7e8cfd1c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:27.187503 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718bbfe3-7176-4a09-b380-19b7e8cfd1c4 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:27.187503 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718bbfe3-7176-4a09-b380-19b7e8cfd1c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:27.187503 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718bbfe3-7176-4a09-b380-19b7e8cfd1c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:27.196059 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718bbfe3-7176-4a09-b380-19b7e8cfd1c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:27.197760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718bbfe3-7176-4a09-b380-19b7e8cfd1c4 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:27.207540 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1152/2263] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:27.473177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3571cfc3-bd5f-44f1-bac0-df644fbcba88 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:27.475559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3571cfc3-bd5f-44f1-bac0-df644fbcba88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:27.488092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3571cfc3-bd5f-44f1-bac0-df644fbcba88 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:27.493184 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4733b005-43f7-4134-b29e-06d8c6d73718 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:27.495295 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4733b005-43f7-4134-b29e-06d8c6d73718 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:27.495599 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4733b005-43f7-4134-b29e-06d8c6d73718 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:27.495874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4733b005-43f7-4134-b29e-06d8c6d73718 admin admin] PATH_INFO: `/v3/projects/f9ad83b53a08431696ba4fd2f6d519eb` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:27.497053 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4733b005-43f7-4134-b29e-06d8c6d73718 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f9ad83b53a08431696ba4fd2f6d519eb)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:27.498995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4733b005-43f7-4134-b29e-06d8c6d73718 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:27.549724 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4733b005-43f7-4134-b29e-06d8c6d73718 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f9ad83b53a08431696ba4fd2f6d519eb', 'request_id': 'req-4733b005-43f7-4134-b29e-06d8c6d73718'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:27.557820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4733b005-43f7-4134-b29e-06d8c6d73718 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f9ad83b53a08431696ba4fd2f6d519eb', 'request_id': 'req-4733b005-43f7-4134-b29e-06d8c6d73718'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:27.565122 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4733b005-43f7-4134-b29e-06d8c6d73718 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f9ad83b53a08431696ba4fd2f6d519eb', 'request_id': 'req-4733b005-43f7-4134-b29e-06d8c6d73718'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:27.577945 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4733b005-43f7-4134-b29e-06d8c6d73718 admin admin] The token cache is being invalidate because project f9ad83b53a08431696ba4fd2f6d519eb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:04:27.578215 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4733b005-43f7-4134-b29e-06d8c6d73718 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:27.578957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4733b005-43f7-4134-b29e-06d8c6d73718 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:27.580446 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1112/2264] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:04:27 2022] DELETE /identity/v3/projects/f9ad83b53a08431696ba4fd2f6d519eb => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:27.586009 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35699945-229f-4592-81e5-363b02b26e05 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:27.586292 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35699945-229f-4592-81e5-363b02b26e05 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:27.586588 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35699945-229f-4592-81e5-363b02b26e05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:27.606060 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-35699945-229f-4592-81e5-363b02b26e05 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:27.607292 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-35699945-229f-4592-81e5-363b02b26e05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:27.625943 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:27.671444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1153/2265] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:27.679290 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77c41c6-bca1-4df1-adde-7d0b222cc753 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:27.679697 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77c41c6-bca1-4df1-adde-7d0b222cc753 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:27.680061 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77c41c6-bca1-4df1-adde-7d0b222cc753 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:27.701190 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b77c41c6-bca1-4df1-adde-7d0b222cc753 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:27.702422 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b77c41c6-bca1-4df1-adde-7d0b222cc753 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:27.711229 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1113/2266] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 01:04:27.715433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35699945-229f-4592-81e5-363b02b26e05 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:27.725267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35699945-229f-4592-81e5-363b02b26e05 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:27.729877 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021bb1a0-a375-4dd2-8ad5-415b1456619e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:27.730510 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021bb1a0-a375-4dd2-8ad5-415b1456619e admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:27.730793 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021bb1a0-a375-4dd2-8ad5-415b1456619e admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:27.731104 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021bb1a0-a375-4dd2-8ad5-415b1456619e admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:27.731756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021bb1a0-a375-4dd2-8ad5-415b1456619e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:04:27.732065 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021bb1a0-a375-4dd2-8ad5-415b1456619e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:27.733673 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021bb1a0-a375-4dd2-8ad5-415b1456619e admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:27.746709 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1154/2267] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:04:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:04:27.751486 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77c41c6-bca1-4df1-adde-7d0b222cc753 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:27.761674 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77c41c6-bca1-4df1-adde-7d0b222cc753 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:27.766569 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45ce42f-a553-4bf7-8d4f-9abec69587fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:27.767428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45ce42f-a553-4bf7-8d4f-9abec69587fc admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:27.767739 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45ce42f-a553-4bf7-8d4f-9abec69587fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:27.768034 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45ce42f-a553-4bf7-8d4f-9abec69587fc admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:27.768829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45ce42f-a553-4bf7-8d4f-9abec69587fc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:27.771466 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45ce42f-a553-4bf7-8d4f-9abec69587fc admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:27.800318 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1114/2268] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:04:27 2022] POST /identity/v3/projects => generated 373 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:04:27.804991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021bb1a0-a375-4dd2-8ad5-415b1456619e admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:27.815033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021bb1a0-a375-4dd2-8ad5-415b1456619e admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:27.876096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c773df-7fea-4f7d-bd79-893f3da90772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:27.876833 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c773df-7fea-4f7d-bd79-893f3da90772 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:27.877004 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c773df-7fea-4f7d-bd79-893f3da90772 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:27.877282 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c773df-7fea-4f7d-bd79-893f3da90772 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:27.877942 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c773df-7fea-4f7d-bd79-893f3da90772 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:27.879996 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c773df-7fea-4f7d-bd79-893f3da90772 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:27.906365 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1155/2269] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:04:27 2022] POST /identity/v3/users => generated 287 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 01:04:27.911250 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45ce42f-a553-4bf7-8d4f-9abec69587fc admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:27.923469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45ce42f-a553-4bf7-8d4f-9abec69587fc admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:27.928986 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b6a1dd-2239-4e68-88ac-75ef07d3af88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:27.929616 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b6a1dd-2239-4e68-88ac-75ef07d3af88 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:27.929898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b6a1dd-2239-4e68-88ac-75ef07d3af88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:27.930179 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b6a1dd-2239-4e68-88ac-75ef07d3af88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:27.930840 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b6a1dd-2239-4e68-88ac-75ef07d3af88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:27.932572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b6a1dd-2239-4e68-88ac-75ef07d3af88 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:27.945755 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1115/2270] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:27 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:04:27.951008 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c773df-7fea-4f7d-bd79-893f3da90772 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:27.962239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c773df-7fea-4f7d-bd79-893f3da90772 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:27.966995 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831a174a-f95e-4006-a84d-0959e80674c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:27.967620 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831a174a-f95e-4006-a84d-0959e80674c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:27.967903 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831a174a-f95e-4006-a84d-0959e80674c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:27.968181 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831a174a-f95e-4006-a84d-0959e80674c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:27.968836 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831a174a-f95e-4006-a84d-0959e80674c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:27.970829 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831a174a-f95e-4006-a84d-0959e80674c8 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:27.983801 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1156/2271] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:27 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:04:27.990227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b6a1dd-2239-4e68-88ac-75ef07d3af88 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:28.001109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b6a1dd-2239-4e68-88ac-75ef07d3af88 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:28.006990 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9247406-de94-407d-bf7d-ccd93a2d063a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:28.007665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9247406-de94-407d-bf7d-ccd93a2d063a admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:28.007953 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9247406-de94-407d-bf7d-ccd93a2d063a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:28.008227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9247406-de94-407d-bf7d-ccd93a2d063a admin admin] PATH_INFO: `/v3/projects/6a6d4660fe584fd4a1d4fbb73c23778f/users/01485ef057d94356ac4ebdbf973dd2b1/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:28.024369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9247406-de94-407d-bf7d-ccd93a2d063a admin admin] RBAC: Authorizing `identity:create_grant(project_id=6a6d4660fe584fd4a1d4fbb73c23778f, user_id=01485ef057d94356ac4ebdbf973dd2b1, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:28.027125 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9247406-de94-407d-bf7d-ccd93a2d063a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:28.043773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1116/2272] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:04:27 2022] PUT /identity/v3/projects/6a6d4660fe584fd4a1d4fbb73c23778f/users/01485ef057d94356ac4ebdbf973dd2b1/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:28.055359 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aca8604-d2d0-49ca-b8b6-d0a0b8f93faf None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:28.055595 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aca8604-d2d0-49ca-b8b6-d0a0b8f93faf None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:28.055834 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aca8604-d2d0-49ca-b8b6-d0a0b8f93faf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:28.086349 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-3aca8604-d2d0-49ca-b8b6-d0a0b8f93faf None None] MFA Rules not processed for user `01485ef057d94356ac4ebdbf973dd2b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:28.087653 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-3aca8604-d2d0-49ca-b8b6-d0a0b8f93faf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:28.157548 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1157/2273] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:28.169165 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9247406-de94-407d-bf7d-ccd93a2d063a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:28.170920 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a9247406-de94-407d-bf7d-ccd93a2d063a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:28.197099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:28.239344 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9247406-de94-407d-bf7d-ccd93a2d063a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:28.245074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc825362-4985-4ff5-b82a-248348f5b4f4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:28.245723 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc825362-4985-4ff5-b82a-248348f5b4f4 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:28.246002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc825362-4985-4ff5-b82a-248348f5b4f4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:28.246275 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc825362-4985-4ff5-b82a-248348f5b4f4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:28.255376 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc825362-4985-4ff5-b82a-248348f5b4f4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:28.256837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc825362-4985-4ff5-b82a-248348f5b4f4 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:28.266330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1117/2274] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:28 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:31.346409 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aca8604-d2d0-49ca-b8b6-d0a0b8f93faf None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:31.367092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:31.412424 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aca8604-d2d0-49ca-b8b6-d0a0b8f93faf None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:31.416960 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c05ce7b-c909-49b8-897d-22cc484a4c3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:31.417591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c05ce7b-c909-49b8-897d-22cc484a4c3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:31.417871 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c05ce7b-c909-49b8-897d-22cc484a4c3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:31.418146 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c05ce7b-c909-49b8-897d-22cc484a4c3d admin admin] PATH_INFO: `/v3/users/01485ef057d94356ac4ebdbf973dd2b1` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:31.419299 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c05ce7b-c909-49b8-897d-22cc484a4c3d admin admin] RBAC: Authorizing `identity:delete_user(user_id=01485ef057d94356ac4ebdbf973dd2b1)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:31.421232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c05ce7b-c909-49b8-897d-22cc484a4c3d admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:31.468829 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4c05ce7b-c909-49b8-897d-22cc484a4c3d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '01485ef057d94356ac4ebdbf973dd2b1'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:31.469488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4c05ce7b-c909-49b8-897d-22cc484a4c3d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '01485ef057d94356ac4ebdbf973dd2b1'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:31.477759 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4c05ce7b-c909-49b8-897d-22cc484a4c3d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '01485ef057d94356ac4ebdbf973dd2b1'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:31.492025 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4c05ce7b-c909-49b8-897d-22cc484a4c3d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '01485ef057d94356ac4ebdbf973dd2b1'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:31.499086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-4c05ce7b-c909-49b8-897d-22cc484a4c3d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '01485ef057d94356ac4ebdbf973dd2b1'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:31.505432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1158/2275] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:04:31 2022] DELETE /identity/v3/users/01485ef057d94356ac4ebdbf973dd2b1 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:04:31.510156 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72b6cbf-b21e-4e0e-a86c-80c745407459 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:31.510625 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72b6cbf-b21e-4e0e-a86c-80c745407459 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:31.510724 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72b6cbf-b21e-4e0e-a86c-80c745407459 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:31.529466 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c72b6cbf-b21e-4e0e-a86c-80c745407459 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:31.530670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c72b6cbf-b21e-4e0e-a86c-80c745407459 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:31.590157 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1118/2276] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:31.601183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c05ce7b-c909-49b8-897d-22cc484a4c3d admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:31.602715 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4c05ce7b-c909-49b8-897d-22cc484a4c3d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:31.662990 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c05ce7b-c909-49b8-897d-22cc484a4c3d admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:31.666677 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6b12c7-3c43-482e-a6e0-9014fbba0b35 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:31.667321 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6b12c7-3c43-482e-a6e0-9014fbba0b35 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:31.667604 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6b12c7-3c43-482e-a6e0-9014fbba0b35 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:31.667878 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6b12c7-3c43-482e-a6e0-9014fbba0b35 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:31.678916 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6b12c7-3c43-482e-a6e0-9014fbba0b35 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:31.680467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6b12c7-3c43-482e-a6e0-9014fbba0b35 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:31.691233 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1159/2277] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:31.970951 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72b6cbf-b21e-4e0e-a86c-80c745407459 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:31.972925 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c72b6cbf-b21e-4e0e-a86c-80c745407459 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:31.987749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72b6cbf-b21e-4e0e-a86c-80c745407459 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:31.992401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb49fd0e-c4b8-41a5-9b0f-d4912e076b43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:31.993042 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb49fd0e-c4b8-41a5-9b0f-d4912e076b43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:31.993328 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb49fd0e-c4b8-41a5-9b0f-d4912e076b43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:31.994129 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb49fd0e-c4b8-41a5-9b0f-d4912e076b43 admin admin] PATH_INFO: `/v3/projects/6a6d4660fe584fd4a1d4fbb73c23778f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:31.995419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb49fd0e-c4b8-41a5-9b0f-d4912e076b43 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6a6d4660fe584fd4a1d4fbb73c23778f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:31.997419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb49fd0e-c4b8-41a5-9b0f-d4912e076b43 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:32.045608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-eb49fd0e-c4b8-41a5-9b0f-d4912e076b43 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6a6d4660fe584fd4a1d4fbb73c23778f', 'request_id': 'req-eb49fd0e-c4b8-41a5-9b0f-d4912e076b43'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:32.053187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-eb49fd0e-c4b8-41a5-9b0f-d4912e076b43 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6a6d4660fe584fd4a1d4fbb73c23778f', 'request_id': 'req-eb49fd0e-c4b8-41a5-9b0f-d4912e076b43'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:32.059644 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-eb49fd0e-c4b8-41a5-9b0f-d4912e076b43 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6a6d4660fe584fd4a1d4fbb73c23778f', 'request_id': 'req-eb49fd0e-c4b8-41a5-9b0f-d4912e076b43'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:32.070820 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-eb49fd0e-c4b8-41a5-9b0f-d4912e076b43 admin admin] The token cache is being invalidate because project 6a6d4660fe584fd4a1d4fbb73c23778f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:04:32.071132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-eb49fd0e-c4b8-41a5-9b0f-d4912e076b43 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:32.072025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-eb49fd0e-c4b8-41a5-9b0f-d4912e076b43 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:32.073513 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1119/2278] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:04:31 2022] DELETE /identity/v3/projects/6a6d4660fe584fd4a1d4fbb73c23778f => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:32.078498 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655d0be4-2528-410c-a559-eced4600e40e None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:32.078781 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655d0be4-2528-410c-a559-eced4600e40e None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:32.079083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655d0be4-2528-410c-a559-eced4600e40e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:32.100008 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-655d0be4-2528-410c-a559-eced4600e40e None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:32.101151 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-655d0be4-2528-410c-a559-eced4600e40e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:32.136655 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:32.164767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1160/2279] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:32.229627 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bf8b11-544a-415c-a323-a5b757fc708c None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:32.229973 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bf8b11-544a-415c-a323-a5b757fc708c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:32.230258 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bf8b11-544a-415c-a323-a5b757fc708c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:32.250327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-74bf8b11-544a-415c-a323-a5b757fc708c None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:32.251577 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-74bf8b11-544a-415c-a323-a5b757fc708c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:32.260498 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1120/2280] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:32.264909 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655d0be4-2528-410c-a559-eced4600e40e None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:32.276269 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655d0be4-2528-410c-a559-eced4600e40e None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:32.280722 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9071dc6-4848-4cd0-a80e-7c38ea9af80f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:32.281332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9071dc6-4848-4cd0-a80e-7c38ea9af80f admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:32.281621 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9071dc6-4848-4cd0-a80e-7c38ea9af80f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:32.281898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9071dc6-4848-4cd0-a80e-7c38ea9af80f admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:32.282548 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9071dc6-4848-4cd0-a80e-7c38ea9af80f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:04:32.282853 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9071dc6-4848-4cd0-a80e-7c38ea9af80f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:32.284497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9071dc6-4848-4cd0-a80e-7c38ea9af80f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:32.296554 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1161/2281] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:04:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:04:32.301014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74bf8b11-544a-415c-a323-a5b757fc708c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:32.311410 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74bf8b11-544a-415c-a323-a5b757fc708c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:32.316696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d619a0c-d376-4771-b3ba-934a1022e401 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:32.317365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d619a0c-d376-4771-b3ba-934a1022e401 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:32.318366 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d619a0c-d376-4771-b3ba-934a1022e401 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:32.318705 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d619a0c-d376-4771-b3ba-934a1022e401 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:32.319450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d619a0c-d376-4771-b3ba-934a1022e401 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:32.321388 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d619a0c-d376-4771-b3ba-934a1022e401 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:32.350691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1121/2282] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:04:32 2022] POST /identity/v3/projects => generated 377 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:04:32.355096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9071dc6-4848-4cd0-a80e-7c38ea9af80f admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:32.366255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9071dc6-4848-4cd0-a80e-7c38ea9af80f admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:32.438791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3ee01e-c04b-4bd6-ba4b-6d8d95df1fe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:32.439444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3ee01e-c04b-4bd6-ba4b-6d8d95df1fe5 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:32.439728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3ee01e-c04b-4bd6-ba4b-6d8d95df1fe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:32.440006 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3ee01e-c04b-4bd6-ba4b-6d8d95df1fe5 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:32.440637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3ee01e-c04b-4bd6-ba4b-6d8d95df1fe5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:32.442737 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3ee01e-c04b-4bd6-ba4b-6d8d95df1fe5 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:32.464861 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1162/2283] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:04:32 2022] POST /identity/v3/users => generated 289 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:04:32.469182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d619a0c-d376-4771-b3ba-934a1022e401 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:32.480341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d619a0c-d376-4771-b3ba-934a1022e401 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:32.485018 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dbcd32-b062-4c54-8ab1-8b7fab1fb175 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:32.485704 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dbcd32-b062-4c54-8ab1-8b7fab1fb175 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:32.485974 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dbcd32-b062-4c54-8ab1-8b7fab1fb175 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:32.486248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dbcd32-b062-4c54-8ab1-8b7fab1fb175 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:32.486929 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6dbcd32-b062-4c54-8ab1-8b7fab1fb175 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:32.488539 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6dbcd32-b062-4c54-8ab1-8b7fab1fb175 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:32.499090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1122/2284] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:04:32.502910 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3ee01e-c04b-4bd6-ba4b-6d8d95df1fe5 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:32.512935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3ee01e-c04b-4bd6-ba4b-6d8d95df1fe5 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:32.517600 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fe2959-6be8-412d-8479-bb23517a0f1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:32.518198 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fe2959-6be8-412d-8479-bb23517a0f1f admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:32.518506 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fe2959-6be8-412d-8479-bb23517a0f1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:32.518782 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fe2959-6be8-412d-8479-bb23517a0f1f admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:32.521488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75fe2959-6be8-412d-8479-bb23517a0f1f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:32.523330 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75fe2959-6be8-412d-8479-bb23517a0f1f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:32.534728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1163/2285] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:32 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:04:32.539845 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dbcd32-b062-4c54-8ab1-8b7fab1fb175 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:32.550020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dbcd32-b062-4c54-8ab1-8b7fab1fb175 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:32.554971 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074d0172-9868-47e5-884b-58a7cdae2587 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:32.555634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074d0172-9868-47e5-884b-58a7cdae2587 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:32.555917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074d0172-9868-47e5-884b-58a7cdae2587 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:32.556195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074d0172-9868-47e5-884b-58a7cdae2587 admin admin] PATH_INFO: `/v3/projects/2ff5f5f46abe47c3bf0768ff55cd163d/users/525b6cc765094f95817e54dcd7ca8439/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:32.558420 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:32.573198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074d0172-9868-47e5-884b-58a7cdae2587 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ff5f5f46abe47c3bf0768ff55cd163d, user_id=525b6cc765094f95817e54dcd7ca8439, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:32.575962 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074d0172-9868-47e5-884b-58a7cdae2587 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:32.589195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1123/2286] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:04:32 2022] PUT /identity/v3/projects/2ff5f5f46abe47c3bf0768ff55cd163d/users/525b6cc765094f95817e54dcd7ca8439/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:32.599939 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d66813-fcd2-44de-ad9b-78adb249556e None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:32.600225 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d66813-fcd2-44de-ad9b-78adb249556e None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:32.600528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d66813-fcd2-44de-ad9b-78adb249556e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:32.627889 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-91d66813-fcd2-44de-ad9b-78adb249556e None None] MFA Rules not processed for user `525b6cc765094f95817e54dcd7ca8439`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:32.629033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-91d66813-fcd2-44de-ad9b-78adb249556e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:32.693913 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1164/2287] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:32 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:32.698698 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074d0172-9868-47e5-884b-58a7cdae2587 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:32.761823 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074d0172-9868-47e5-884b-58a7cdae2587 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:32.766497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0579655e-7b94-4cb3-8b08-2e7169ebf77c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:32.767140 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0579655e-7b94-4cb3-8b08-2e7169ebf77c admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:32.767425 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0579655e-7b94-4cb3-8b08-2e7169ebf77c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:32.767704 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0579655e-7b94-4cb3-8b08-2e7169ebf77c admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:32.768335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0579655e-7b94-4cb3-8b08-2e7169ebf77c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:32.770257 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0579655e-7b94-4cb3-8b08-2e7169ebf77c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:32.799479 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1124/2288] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:04:32 2022] POST /identity/v3/projects => generated 375 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:04:32.801053 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d66813-fcd2-44de-ad9b-78adb249556e None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:32.812425 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d66813-fcd2-44de-ad9b-78adb249556e None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:32.846661 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:32.873637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96ccdbc-2701-4290-a88f-8c06d5d9e6a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:32.874262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96ccdbc-2701-4290-a88f-8c06d5d9e6a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:32.874544 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96ccdbc-2701-4290-a88f-8c06d5d9e6a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:32.874821 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96ccdbc-2701-4290-a88f-8c06d5d9e6a5 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:32.875505 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96ccdbc-2701-4290-a88f-8c06d5d9e6a5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:32.877470 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96ccdbc-2701-4290-a88f-8c06d5d9e6a5 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:32.904658 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1165/2289] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:04:32 2022] POST /identity/v3/users => generated 288 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:04:32.908761 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0579655e-7b94-4cb3-8b08-2e7169ebf77c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:32.919634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0579655e-7b94-4cb3-8b08-2e7169ebf77c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:32.924336 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf2235f-2620-4e54-b9ab-d3a75e339271 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:32.924938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf2235f-2620-4e54-b9ab-d3a75e339271 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:32.925212 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf2235f-2620-4e54-b9ab-d3a75e339271 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:32.925478 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf2235f-2620-4e54-b9ab-d3a75e339271 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:32.926115 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf2235f-2620-4e54-b9ab-d3a75e339271 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:32.927846 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf2235f-2620-4e54-b9ab-d3a75e339271 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:32.941091 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1125/2290] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:04:32.944452 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96ccdbc-2701-4290-a88f-8c06d5d9e6a5 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:32.955624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96ccdbc-2701-4290-a88f-8c06d5d9e6a5 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:32.961760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1c46c6-482a-4cc9-b116-4aff169c8a5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:32.962416 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1c46c6-482a-4cc9-b116-4aff169c8a5d admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:32.962696 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1c46c6-482a-4cc9-b116-4aff169c8a5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:32.963087 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1c46c6-482a-4cc9-b116-4aff169c8a5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:32.963937 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1c46c6-482a-4cc9-b116-4aff169c8a5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:32.965566 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1c46c6-482a-4cc9-b116-4aff169c8a5d admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:32.977784 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1166/2291] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:32 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:04:32.982231 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf2235f-2620-4e54-b9ab-d3a75e339271 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:32.992472 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf2235f-2620-4e54-b9ab-d3a75e339271 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:32.997141 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781cd426-0d7a-403b-8a0a-122f48592e50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:32.997786 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781cd426-0d7a-403b-8a0a-122f48592e50 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:32.998065 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781cd426-0d7a-403b-8a0a-122f48592e50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:32.998335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781cd426-0d7a-403b-8a0a-122f48592e50 admin admin] PATH_INFO: `/v3/projects/9eda319fd07542cf835907fd935bb29d/users/16a807092f9a474294bce5935ea4d98b/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:33.013898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781cd426-0d7a-403b-8a0a-122f48592e50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9eda319fd07542cf835907fd935bb29d, user_id=16a807092f9a474294bce5935ea4d98b, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:33.016620 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781cd426-0d7a-403b-8a0a-122f48592e50 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:33.030947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1126/2292] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:04:32 2022] PUT /identity/v3/projects/9eda319fd07542cf835907fd935bb29d/users/16a807092f9a474294bce5935ea4d98b/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:33.041781 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f40d9f5-20a0-4d53-a2e8-0b212a62caec None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:33.042140 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f40d9f5-20a0-4d53-a2e8-0b212a62caec None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:33.042425 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f40d9f5-20a0-4d53-a2e8-0b212a62caec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:33.071981 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-7f40d9f5-20a0-4d53-a2e8-0b212a62caec None None] MFA Rules not processed for user `16a807092f9a474294bce5935ea4d98b`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:33.073381 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-7f40d9f5-20a0-4d53-a2e8-0b212a62caec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:33.137606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1167/2293] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:33 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:33.142225 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781cd426-0d7a-403b-8a0a-122f48592e50 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:33.205146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781cd426-0d7a-403b-8a0a-122f48592e50 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:33.210661 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dc0060-d673-4b74-9238-ccac031290f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:33.211331 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dc0060-d673-4b74-9238-ccac031290f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:33.211619 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dc0060-d673-4b74-9238-ccac031290f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:33.211895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dc0060-d673-4b74-9238-ccac031290f8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:33.212542 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dc0060-d673-4b74-9238-ccac031290f8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:33.214485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dc0060-d673-4b74-9238-ccac031290f8 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:33.242241 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1127/2294] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:04:33 2022] POST /identity/v3/projects => generated 375 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 01:04:33.246187 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f40d9f5-20a0-4d53-a2e8-0b212a62caec None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:33.257634 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f40d9f5-20a0-4d53-a2e8-0b212a62caec None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:33.313979 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff937687-8b2c-4206-a8c0-3e2f3c1a87bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:33.314596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff937687-8b2c-4206-a8c0-3e2f3c1a87bb admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:33.314896 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff937687-8b2c-4206-a8c0-3e2f3c1a87bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:33.315178 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff937687-8b2c-4206-a8c0-3e2f3c1a87bb admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:33.315976 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff937687-8b2c-4206-a8c0-3e2f3c1a87bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:33.318403 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff937687-8b2c-4206-a8c0-3e2f3c1a87bb admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:33.342560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1168/2295] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:04:33 2022] POST /identity/v3/users => generated 294 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:04:33.343723 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dc0060-d673-4b74-9238-ccac031290f8 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:33.353965 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dc0060-d673-4b74-9238-ccac031290f8 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:33.358617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14eb2d8-edfc-47bb-9836-d7c152b363d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:33.359248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14eb2d8-edfc-47bb-9836-d7c152b363d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:33.359529 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14eb2d8-edfc-47bb-9836-d7c152b363d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:33.359805 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14eb2d8-edfc-47bb-9836-d7c152b363d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:33.360448 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14eb2d8-edfc-47bb-9836-d7c152b363d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:33.362047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14eb2d8-edfc-47bb-9836-d7c152b363d2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:33.373513 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1128/2296] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:33 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 01:04:33.377734 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff937687-8b2c-4206-a8c0-3e2f3c1a87bb admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:33.388510 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff937687-8b2c-4206-a8c0-3e2f3c1a87bb admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:33.393210 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d8665e-d6a6-4ba3-bc1b-ccac680477d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:33.393871 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d8665e-d6a6-4ba3-bc1b-ccac680477d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:33.394156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d8665e-d6a6-4ba3-bc1b-ccac680477d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:33.394435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d8665e-d6a6-4ba3-bc1b-ccac680477d3 admin admin] PATH_INFO: `/v3/projects/0faf17f64b0a43769d0a48faa991cdc2/users/a1cb29f3f4dd4be3982e1be397fbdbd0/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:33.410173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d8665e-d6a6-4ba3-bc1b-ccac680477d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0faf17f64b0a43769d0a48faa991cdc2, user_id=a1cb29f3f4dd4be3982e1be397fbdbd0, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:33.413047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d8665e-d6a6-4ba3-bc1b-ccac680477d3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:33.428027 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1169/2297] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:04:33 2022] PUT /identity/v3/projects/0faf17f64b0a43769d0a48faa991cdc2/users/a1cb29f3f4dd4be3982e1be397fbdbd0/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:04:33.432192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14eb2d8-edfc-47bb-9836-d7c152b363d2 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:33.443203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14eb2d8-edfc-47bb-9836-d7c152b363d2 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:33.447390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:33.505910 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d445c4-6bce-4404-8c19-fcb7c00c9360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:33.506549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d445c4-6bce-4404-8c19-fcb7c00c9360 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:33.506940 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d445c4-6bce-4404-8c19-fcb7c00c9360 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:33.507222 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d445c4-6bce-4404-8c19-fcb7c00c9360 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:33.507866 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13d445c4-6bce-4404-8c19-fcb7c00c9360 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:33.509486 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13d445c4-6bce-4404-8c19-fcb7c00c9360 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:33.519659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1129/2298] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:33 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 01:04:33.524209 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d8665e-d6a6-4ba3-bc1b-ccac680477d3 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:33.534835 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d8665e-d6a6-4ba3-bc1b-ccac680477d3 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:33.540293 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5037c3-b34e-4bb5-bfb9-b3293be999a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:33.540963 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5037c3-b34e-4bb5-bfb9-b3293be999a0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:33.541248 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5037c3-b34e-4bb5-bfb9-b3293be999a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:33.541667 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5037c3-b34e-4bb5-bfb9-b3293be999a0 admin admin] PATH_INFO: `/v3/projects/0faf17f64b0a43769d0a48faa991cdc2/users/a1cb29f3f4dd4be3982e1be397fbdbd0/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:33.543723 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5037c3-b34e-4bb5-bfb9-b3293be999a0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0faf17f64b0a43769d0a48faa991cdc2, user_id=a1cb29f3f4dd4be3982e1be397fbdbd0, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:33.546415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5037c3-b34e-4bb5-bfb9-b3293be999a0 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:33.549744 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:33.559880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1170/2299] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:04:33 2022] PUT /identity/v3/projects/0faf17f64b0a43769d0a48faa991cdc2/users/a1cb29f3f4dd4be3982e1be397fbdbd0/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:04:33.571843 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cf77ec-ad1b-457a-834d-396245a325c3 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:33.572419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cf77ec-ad1b-457a-834d-396245a325c3 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:33.572419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cf77ec-ad1b-457a-834d-396245a325c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:33.608927 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-38cf77ec-ad1b-457a-834d-396245a325c3 None None] MFA Rules not processed for user `a1cb29f3f4dd4be3982e1be397fbdbd0`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:33.610142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-38cf77ec-ad1b-457a-834d-396245a325c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:33.678450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1130/2300] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:33 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:33.689796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5037c3-b34e-4bb5-bfb9-b3293be999a0 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:33.691578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6d5037c3-b34e-4bb5-bfb9-b3293be999a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:33.750750 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5037c3-b34e-4bb5-bfb9-b3293be999a0 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:33.754353 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfe3f15-98ca-4f9f-a42d-82e1ba10dfb3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:33.755044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfe3f15-98ca-4f9f-a42d-82e1ba10dfb3 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:33.755325 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfe3f15-98ca-4f9f-a42d-82e1ba10dfb3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:33.755631 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfe3f15-98ca-4f9f-a42d-82e1ba10dfb3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:33.813601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfe3f15-98ca-4f9f-a42d-82e1ba10dfb3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:33.815065 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfe3f15-98ca-4f9f-a42d-82e1ba10dfb3 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:33.824243 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1171/2301] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:33 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:35.353679 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cf77ec-ad1b-457a-834d-396245a325c3 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:35.355198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-38cf77ec-ad1b-457a-834d-396245a325c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:35.368308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cf77ec-ad1b-457a-834d-396245a325c3 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:35.372106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d72bc0-dff2-4fd1-bd99-295056c1449a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:35.372719 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d72bc0-dff2-4fd1-bd99-295056c1449a service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:35.373133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d72bc0-dff2-4fd1-bd99-295056c1449a service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:35.373413 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d72bc0-dff2-4fd1-bd99-295056c1449a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:35.432033 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d72bc0-dff2-4fd1-bd99-295056c1449a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:35.433508 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d72bc0-dff2-4fd1-bd99-295056c1449a service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:35.442767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1131/2302] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:35 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:36.019924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfe3f15-98ca-4f9f-a42d-82e1ba10dfb3 service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:36.029341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfe3f15-98ca-4f9f-a42d-82e1ba10dfb3 service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:36.033025 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c09d3a6-3892-4758-b606-751354aa5f3f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:36.033631 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c09d3a6-3892-4758-b606-751354aa5f3f service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:36.033913 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c09d3a6-3892-4758-b606-751354aa5f3f service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:36.034185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c09d3a6-3892-4758-b606-751354aa5f3f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:36.043518 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c09d3a6-3892-4758-b606-751354aa5f3f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:36.044945 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c09d3a6-3892-4758-b606-751354aa5f3f service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:36.055485 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1172/2303] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:36 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:40.069719 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d72bc0-dff2-4fd1-bd99-295056c1449a service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:40.107478 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:40.152035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d72bc0-dff2-4fd1-bd99-295056c1449a service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:40.161827 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6a1752-4ee4-4dd9-9866-cd4b44c7344b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:40.162640 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6a1752-4ee4-4dd9-9866-cd4b44c7344b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:40.163114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6a1752-4ee4-4dd9-9866-cd4b44c7344b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:40.163618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6a1752-4ee4-4dd9-9866-cd4b44c7344b admin admin] PATH_INFO: `/v3/users/525b6cc765094f95817e54dcd7ca8439` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:40.165179 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6a1752-4ee4-4dd9-9866-cd4b44c7344b admin admin] RBAC: Authorizing `identity:delete_user(user_id=525b6cc765094f95817e54dcd7ca8439)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:40.167367 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6a1752-4ee4-4dd9-9866-cd4b44c7344b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:40.237429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cb6a1752-4ee4-4dd9-9866-cd4b44c7344b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '525b6cc765094f95817e54dcd7ca8439'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:40.238216 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cb6a1752-4ee4-4dd9-9866-cd4b44c7344b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '525b6cc765094f95817e54dcd7ca8439'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:40.247527 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cb6a1752-4ee4-4dd9-9866-cd4b44c7344b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '525b6cc765094f95817e54dcd7ca8439'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:40.248221 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cb6a1752-4ee4-4dd9-9866-cd4b44c7344b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '525b6cc765094f95817e54dcd7ca8439'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:40.263486 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cb6a1752-4ee4-4dd9-9866-cd4b44c7344b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '525b6cc765094f95817e54dcd7ca8439'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:40.284957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1132/2304] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:04:40 2022] DELETE /identity/v3/users/525b6cc765094f95817e54dcd7ca8439 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:40.290465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950c8b7c-9d6b-4c87-b6d0-7177c3cfc02b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:40.291902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950c8b7c-9d6b-4c87-b6d0-7177c3cfc02b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950c8b7c-9d6b-4c87-b6d0-7177c3cfc02b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:40.330096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-950c8b7c-9d6b-4c87-b6d0-7177c3cfc02b None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:40.331470 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-950c8b7c-9d6b-4c87-b6d0-7177c3cfc02b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:40.398003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1173/2305] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:40.407599 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6a1752-4ee4-4dd9-9866-cd4b44c7344b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:40.409016 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cb6a1752-4ee4-4dd9-9866-cd4b44c7344b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:40.473108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6a1752-4ee4-4dd9-9866-cd4b44c7344b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:40.476911 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e959614-b20e-4f03-8b35-07e0807bb442 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:40.477556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e959614-b20e-4f03-8b35-07e0807bb442 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:40.477840 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e959614-b20e-4f03-8b35-07e0807bb442 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:40.478113 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e959614-b20e-4f03-8b35-07e0807bb442 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:40.488360 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e959614-b20e-4f03-8b35-07e0807bb442 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:40.490004 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e959614-b20e-4f03-8b35-07e0807bb442 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:40.501805 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1133/2306] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:40.925905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950c8b7c-9d6b-4c87-b6d0-7177c3cfc02b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:40.927630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-950c8b7c-9d6b-4c87-b6d0-7177c3cfc02b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:40.945580 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950c8b7c-9d6b-4c87-b6d0-7177c3cfc02b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:40.950661 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1e05c9-5241-4fd8-aab9-47be9200cd72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:40.951351 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1e05c9-5241-4fd8-aab9-47be9200cd72 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:40.951634 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1e05c9-5241-4fd8-aab9-47be9200cd72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:40.951912 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1e05c9-5241-4fd8-aab9-47be9200cd72 admin admin] PATH_INFO: `/v3/projects/2ff5f5f46abe47c3bf0768ff55cd163d` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:40.953427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1e05c9-5241-4fd8-aab9-47be9200cd72 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2ff5f5f46abe47c3bf0768ff55cd163d)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:40.955525 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1e05c9-5241-4fd8-aab9-47be9200cd72 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:41.002248 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:41.017874 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3f1e05c9-5241-4fd8-aab9-47be9200cd72 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2ff5f5f46abe47c3bf0768ff55cd163d', 'request_id': 'req-3f1e05c9-5241-4fd8-aab9-47be9200cd72'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.027489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3f1e05c9-5241-4fd8-aab9-47be9200cd72 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2ff5f5f46abe47c3bf0768ff55cd163d', 'request_id': 'req-3f1e05c9-5241-4fd8-aab9-47be9200cd72'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.035741 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3f1e05c9-5241-4fd8-aab9-47be9200cd72 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2ff5f5f46abe47c3bf0768ff55cd163d', 'request_id': 'req-3f1e05c9-5241-4fd8-aab9-47be9200cd72'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.051735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3f1e05c9-5241-4fd8-aab9-47be9200cd72 admin admin] The token cache is being invalidate because project 2ff5f5f46abe47c3bf0768ff55cd163d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:04:41.052024 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3f1e05c9-5241-4fd8-aab9-47be9200cd72 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.052900 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3f1e05c9-5241-4fd8-aab9-47be9200cd72 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.054559 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1174/2307] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:04:40 2022] DELETE /identity/v3/projects/2ff5f5f46abe47c3bf0768ff55cd163d => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:41.059984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e959614-b20e-4f03-8b35-07e0807bb442 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:41.071749 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e959614-b20e-4f03-8b35-07e0807bb442 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:41.117882 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:41.131775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f855496-72bb-484d-84f1-fcbae2659100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:41.132413 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f855496-72bb-484d-84f1-fcbae2659100 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:41.132700 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f855496-72bb-484d-84f1-fcbae2659100 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:41.133032 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f855496-72bb-484d-84f1-fcbae2659100 admin admin] PATH_INFO: `/v3/users/16a807092f9a474294bce5935ea4d98b` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:41.134233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f855496-72bb-484d-84f1-fcbae2659100 admin admin] RBAC: Authorizing `identity:delete_user(user_id=16a807092f9a474294bce5935ea4d98b)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:41.136334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f855496-72bb-484d-84f1-fcbae2659100 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:41.185104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5f855496-72bb-484d-84f1-fcbae2659100 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '16a807092f9a474294bce5935ea4d98b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.185751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5f855496-72bb-484d-84f1-fcbae2659100 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '16a807092f9a474294bce5935ea4d98b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.193974 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5f855496-72bb-484d-84f1-fcbae2659100 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '16a807092f9a474294bce5935ea4d98b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.194602 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5f855496-72bb-484d-84f1-fcbae2659100 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '16a807092f9a474294bce5935ea4d98b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.209870 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-5f855496-72bb-484d-84f1-fcbae2659100 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '16a807092f9a474294bce5935ea4d98b'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.223748 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1134/2308] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:04:41 2022] DELETE /identity/v3/users/16a807092f9a474294bce5935ea4d98b => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:41.505384 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1e05c9-5241-4fd8-aab9-47be9200cd72 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:41.508899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-3f1e05c9-5241-4fd8-aab9-47be9200cd72 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:41.527816 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1e05c9-5241-4fd8-aab9-47be9200cd72 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:41.585668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ed03b4-aa17-4a48-b95e-571878f44b6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:41.586314 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ed03b4-aa17-4a48-b95e-571878f44b6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:41.586600 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ed03b4-aa17-4a48-b95e-571878f44b6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:41.586904 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ed03b4-aa17-4a48-b95e-571878f44b6f admin admin] PATH_INFO: `/v3/projects/9eda319fd07542cf835907fd935bb29d` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:41.588044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ed03b4-aa17-4a48-b95e-571878f44b6f admin admin] RBAC: Authorizing `identity:delete_project(project_id=9eda319fd07542cf835907fd935bb29d)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:41.589972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ed03b4-aa17-4a48-b95e-571878f44b6f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:41.637654 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-25ed03b4-aa17-4a48-b95e-571878f44b6f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9eda319fd07542cf835907fd935bb29d', 'request_id': 'req-25ed03b4-aa17-4a48-b95e-571878f44b6f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.645132 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-25ed03b4-aa17-4a48-b95e-571878f44b6f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9eda319fd07542cf835907fd935bb29d', 'request_id': 'req-25ed03b4-aa17-4a48-b95e-571878f44b6f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.650900 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-25ed03b4-aa17-4a48-b95e-571878f44b6f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9eda319fd07542cf835907fd935bb29d', 'request_id': 'req-25ed03b4-aa17-4a48-b95e-571878f44b6f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.662286 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-25ed03b4-aa17-4a48-b95e-571878f44b6f admin admin] The token cache is being invalidate because project 9eda319fd07542cf835907fd935bb29d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:04:41.662581 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-25ed03b4-aa17-4a48-b95e-571878f44b6f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.663252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-25ed03b4-aa17-4a48-b95e-571878f44b6f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.664746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1175/2309] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:04:41 2022] DELETE /identity/v3/projects/9eda319fd07542cf835907fd935bb29d => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:41.668831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f855496-72bb-484d-84f1-fcbae2659100 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:41.670231 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-5f855496-72bb-484d-84f1-fcbae2659100 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:41.683466 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f855496-72bb-484d-84f1-fcbae2659100 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:41.742218 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3e1c4b-f620-4680-83da-2854a68a8503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:41.742840 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3e1c4b-f620-4680-83da-2854a68a8503 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:41.743151 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3e1c4b-f620-4680-83da-2854a68a8503 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:41.743531 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3e1c4b-f620-4680-83da-2854a68a8503 admin admin] PATH_INFO: `/v3/users/a1cb29f3f4dd4be3982e1be397fbdbd0` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:41.744679 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3e1c4b-f620-4680-83da-2854a68a8503 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a1cb29f3f4dd4be3982e1be397fbdbd0)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:41.746604 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3e1c4b-f620-4680-83da-2854a68a8503 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:41.792357 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7a3e1c4b-f620-4680-83da-2854a68a8503 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a1cb29f3f4dd4be3982e1be397fbdbd0'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.793009 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7a3e1c4b-f620-4680-83da-2854a68a8503 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a1cb29f3f4dd4be3982e1be397fbdbd0'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.800127 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7a3e1c4b-f620-4680-83da-2854a68a8503 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a1cb29f3f4dd4be3982e1be397fbdbd0'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.800804 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7a3e1c4b-f620-4680-83da-2854a68a8503 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a1cb29f3f4dd4be3982e1be397fbdbd0'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.814362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:41.816164 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-7a3e1c4b-f620-4680-83da-2854a68a8503 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a1cb29f3f4dd4be3982e1be397fbdbd0'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:41.831138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1135/2310] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:04:41 2022] DELETE /identity/v3/users/a1cb29f3f4dd4be3982e1be397fbdbd0 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:42.138390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ed03b4-aa17-4a48-b95e-571878f44b6f admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:42.139883 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-25ed03b4-aa17-4a48-b95e-571878f44b6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:42.153700 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ed03b4-aa17-4a48-b95e-571878f44b6f admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:42.237782 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:42.254410 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d576c8e1-373f-4455-8b79-23aa8afe0714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:42.255102 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d576c8e1-373f-4455-8b79-23aa8afe0714 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:42.255392 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d576c8e1-373f-4455-8b79-23aa8afe0714 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:42.255757 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d576c8e1-373f-4455-8b79-23aa8afe0714 admin admin] PATH_INFO: `/v3/projects/0faf17f64b0a43769d0a48faa991cdc2` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:42.256826 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d576c8e1-373f-4455-8b79-23aa8afe0714 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0faf17f64b0a43769d0a48faa991cdc2)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:42.259149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d576c8e1-373f-4455-8b79-23aa8afe0714 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:42.312043 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d576c8e1-373f-4455-8b79-23aa8afe0714 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0faf17f64b0a43769d0a48faa991cdc2', 'request_id': 'req-d576c8e1-373f-4455-8b79-23aa8afe0714'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:42.320916 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d576c8e1-373f-4455-8b79-23aa8afe0714 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0faf17f64b0a43769d0a48faa991cdc2', 'request_id': 'req-d576c8e1-373f-4455-8b79-23aa8afe0714'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:42.329620 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d576c8e1-373f-4455-8b79-23aa8afe0714 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0faf17f64b0a43769d0a48faa991cdc2', 'request_id': 'req-d576c8e1-373f-4455-8b79-23aa8afe0714'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:42.350003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d576c8e1-373f-4455-8b79-23aa8afe0714 admin admin] The token cache is being invalidate because project 0faf17f64b0a43769d0a48faa991cdc2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:04:42.350003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d576c8e1-373f-4455-8b79-23aa8afe0714 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:42.350003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-d576c8e1-373f-4455-8b79-23aa8afe0714 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:42.350003 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1176/2311] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:04:42 2022] DELETE /identity/v3/projects/0faf17f64b0a43769d0a48faa991cdc2 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:42.354113 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d7a4ab-fcf4-49af-b0a5-c3f349c0d757 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:42.354613 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d7a4ab-fcf4-49af-b0a5-c3f349c0d757 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:42.354714 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d7a4ab-fcf4-49af-b0a5-c3f349c0d757 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:42.374245 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-26d7a4ab-fcf4-49af-b0a5-c3f349c0d757 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:42.375468 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-26d7a4ab-fcf4-49af-b0a5-c3f349c0d757 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:42.434631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1136/2312] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:42.443365 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a9dfde-a86c-430b-a88d-48c407b6dbed None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:42.443659 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a9dfde-a86c-430b-a88d-48c407b6dbed None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:42.443939 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a9dfde-a86c-430b-a88d-48c407b6dbed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:42.465256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-65a9dfde-a86c-430b-a88d-48c407b6dbed None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:42.466489 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-65a9dfde-a86c-430b-a88d-48c407b6dbed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:42.474692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1177/2313] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:42.479046 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d7a4ab-fcf4-49af-b0a5-c3f349c0d757 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:42.489567 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d7a4ab-fcf4-49af-b0a5-c3f349c0d757 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:42.494624 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ac63ab-c46d-4270-baaa-62500dac4845 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:42.495430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ac63ab-c46d-4270-baaa-62500dac4845 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:42.495764 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ac63ab-c46d-4270-baaa-62500dac4845 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:42.496071 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ac63ab-c46d-4270-baaa-62500dac4845 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:42.496768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ac63ab-c46d-4270-baaa-62500dac4845 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:04:42.497072 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ac63ab-c46d-4270-baaa-62500dac4845 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:42.498720 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ac63ab-c46d-4270-baaa-62500dac4845 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:42.512558 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1137/2314] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:04:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:04:42.517118 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a9dfde-a86c-430b-a88d-48c407b6dbed None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:42.526765 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a9dfde-a86c-430b-a88d-48c407b6dbed None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:42.532378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e10afa-7c9e-4059-bb79-f8cdb04763d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:42.533088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e10afa-7c9e-4059-bb79-f8cdb04763d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:42.533302 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e10afa-7c9e-4059-bb79-f8cdb04763d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:42.533578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e10afa-7c9e-4059-bb79-f8cdb04763d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:42.534251 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e10afa-7c9e-4059-bb79-f8cdb04763d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:42.536848 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e10afa-7c9e-4059-bb79-f8cdb04763d2 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:42.565791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1178/2315] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:04:42 2022] POST /identity/v3/projects => generated 399 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:04:42.570017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ac63ab-c46d-4270-baaa-62500dac4845 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:42.580158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ac63ab-c46d-4270-baaa-62500dac4845 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:42.636923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1499cb22-f7f7-435a-98f1-3e2bdfec2120 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:42.637532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1499cb22-f7f7-435a-98f1-3e2bdfec2120 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:42.637818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1499cb22-f7f7-435a-98f1-3e2bdfec2120 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:42.638095 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1499cb22-f7f7-435a-98f1-3e2bdfec2120 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:42.638720 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1499cb22-f7f7-435a-98f1-3e2bdfec2120 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:42.640691 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1499cb22-f7f7-435a-98f1-3e2bdfec2120 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:42.662950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1138/2316] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:04:42 2022] POST /identity/v3/users => generated 300 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:04:42.667121 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e10afa-7c9e-4059-bb79-f8cdb04763d2 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:42.678094 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e10afa-7c9e-4059-bb79-f8cdb04763d2 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:42.682725 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8eade4-7719-4e11-97e7-b018b60e69f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:42.683362 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8eade4-7719-4e11-97e7-b018b60e69f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:42.683757 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8eade4-7719-4e11-97e7-b018b60e69f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:42.684035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8eade4-7719-4e11-97e7-b018b60e69f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:42.684674 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8eade4-7719-4e11-97e7-b018b60e69f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:42.686266 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8eade4-7719-4e11-97e7-b018b60e69f7 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:42.696647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1179/2317] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:04:42.700660 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1499cb22-f7f7-435a-98f1-3e2bdfec2120 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:42.710848 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1499cb22-f7f7-435a-98f1-3e2bdfec2120 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:42.715791 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1218354c-509e-4072-9236-ed6c7cdb086f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:42.716397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1218354c-509e-4072-9236-ed6c7cdb086f admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:42.716684 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1218354c-509e-4072-9236-ed6c7cdb086f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:42.716963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1218354c-509e-4072-9236-ed6c7cdb086f admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:42.717619 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1218354c-509e-4072-9236-ed6c7cdb086f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:42.719281 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1218354c-509e-4072-9236-ed6c7cdb086f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:42.735041 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1139/2318] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:42 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:04:42.741293 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8eade4-7719-4e11-97e7-b018b60e69f7 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:42.752141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8eade4-7719-4e11-97e7-b018b60e69f7 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:42.756837 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb45fa54-059e-43d8-bf99-3299a45050b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:42.757497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb45fa54-059e-43d8-bf99-3299a45050b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:42.757812 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb45fa54-059e-43d8-bf99-3299a45050b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:42.758094 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb45fa54-059e-43d8-bf99-3299a45050b9 admin admin] PATH_INFO: `/v3/projects/290958d9e5034a9bb65cadd610860cc8/users/3a4e6e422e77495db2523ce52ca5a13c/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:42.773121 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb45fa54-059e-43d8-bf99-3299a45050b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=290958d9e5034a9bb65cadd610860cc8, user_id=3a4e6e422e77495db2523ce52ca5a13c, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:42.775850 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb45fa54-059e-43d8-bf99-3299a45050b9 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:42.791185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1180/2319] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:04:42 2022] PUT /identity/v3/projects/290958d9e5034a9bb65cadd610860cc8/users/3a4e6e422e77495db2523ce52ca5a13c/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:42.803989 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117e782f-5d24-4e53-b243-930b763ea741 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:42.804274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117e782f-5d24-4e53-b243-930b763ea741 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:42.804547 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117e782f-5d24-4e53-b243-930b763ea741 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:42.833582 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-117e782f-5d24-4e53-b243-930b763ea741 None None] MFA Rules not processed for user `3a4e6e422e77495db2523ce52ca5a13c`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:42.835389 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-117e782f-5d24-4e53-b243-930b763ea741 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:42.877136 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:42.901941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1140/2320] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:42 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:42.914028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb45fa54-059e-43d8-bf99-3299a45050b9 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:42.915918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-eb45fa54-059e-43d8-bf99-3299a45050b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:42.975664 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb45fa54-059e-43d8-bf99-3299a45050b9 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:42.979433 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fcc411-a672-4505-8a0e-2af2f3ba7487 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:42.980046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fcc411-a672-4505-8a0e-2af2f3ba7487 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:42.980325 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fcc411-a672-4505-8a0e-2af2f3ba7487 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:42.980595 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fcc411-a672-4505-8a0e-2af2f3ba7487 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:42.989534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fcc411-a672-4505-8a0e-2af2f3ba7487 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:42.991004 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fcc411-a672-4505-8a0e-2af2f3ba7487 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:42.999912 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1181/2321] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:42 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:43.485373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117e782f-5d24-4e53-b243-930b763ea741 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:43.487493 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-117e782f-5d24-4e53-b243-930b763ea741 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:43.554367 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117e782f-5d24-4e53-b243-930b763ea741 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:43.558715 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b52285-1cb6-43f1-a469-1df8e5e4a18c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:43.559445 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b52285-1cb6-43f1-a469-1df8e5e4a18c service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:43.560539 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b52285-1cb6-43f1-a469-1df8e5e4a18c service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:43.560539 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b52285-1cb6-43f1-a469-1df8e5e4a18c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:43.562016 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-05b52285-1cb6-43f1-a469-1df8e5e4a18c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:43.636853 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b52285-1cb6-43f1-a469-1df8e5e4a18c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:43.638338 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b52285-1cb6-43f1-a469-1df8e5e4a18c service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:43.650048 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1141/2322] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:46.993104 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fcc411-a672-4505-8a0e-2af2f3ba7487 service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:46.994793 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-92fcc411-a672-4505-8a0e-2af2f3ba7487 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:47.015314 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fcc411-a672-4505-8a0e-2af2f3ba7487 service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:47.022346 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd538a0-f6c2-467c-b2ce-5a3965cf19a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:47.022897 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd538a0-f6c2-467c-b2ce-5a3965cf19a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:47.023285 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd538a0-f6c2-467c-b2ce-5a3965cf19a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:47.023752 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd538a0-f6c2-467c-b2ce-5a3965cf19a6 admin admin] PATH_INFO: `/v3/users/ccece85a85fb48c4887b52c96c5bc642` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:47.025185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd538a0-f6c2-467c-b2ce-5a3965cf19a6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ccece85a85fb48c4887b52c96c5bc642)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:47.028032 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd538a0-f6c2-467c-b2ce-5a3965cf19a6 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:47.045314 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=8,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:47.088107 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-bdd538a0-f6c2-467c-b2ce-5a3965cf19a6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ccece85a85fb48c4887b52c96c5bc642'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:47.088770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-bdd538a0-f6c2-467c-b2ce-5a3965cf19a6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ccece85a85fb48c4887b52c96c5bc642'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:47.096527 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-bdd538a0-f6c2-467c-b2ce-5a3965cf19a6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ccece85a85fb48c4887b52c96c5bc642'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:47.115719 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-bdd538a0-f6c2-467c-b2ce-5a3965cf19a6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ccece85a85fb48c4887b52c96c5bc642'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:47.123208 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-bdd538a0-f6c2-467c-b2ce-5a3965cf19a6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ccece85a85fb48c4887b52c96c5bc642'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:47.129647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1182/2323] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:04:46 2022] DELETE /identity/v3/users/ccece85a85fb48c4887b52c96c5bc642 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:47.305984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b52285-1cb6-43f1-a469-1df8e5e4a18c service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:47.316015 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b52285-1cb6-43f1-a469-1df8e5e4a18c service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:47.375534 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c4826f-4024-42b5-be69-b109a1b572d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:47.376862 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c4826f-4024-42b5-be69-b109a1b572d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:47.376862 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c4826f-4024-42b5-be69-b109a1b572d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:47.376862 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c4826f-4024-42b5-be69-b109a1b572d6 admin admin] PATH_INFO: `/v3/projects/cca355ef4a294ecd88cc299b7e0ca9d8` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:47.377886 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c4826f-4024-42b5-be69-b109a1b572d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cca355ef4a294ecd88cc299b7e0ca9d8)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:47.380500 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c4826f-4024-42b5-be69-b109a1b572d6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:47.423600 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:47.437588 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-74c4826f-4024-42b5-be69-b109a1b572d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cca355ef4a294ecd88cc299b7e0ca9d8', 'request_id': 'req-74c4826f-4024-42b5-be69-b109a1b572d6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:47.446835 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-74c4826f-4024-42b5-be69-b109a1b572d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cca355ef4a294ecd88cc299b7e0ca9d8', 'request_id': 'req-74c4826f-4024-42b5-be69-b109a1b572d6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:47.453838 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-74c4826f-4024-42b5-be69-b109a1b572d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cca355ef4a294ecd88cc299b7e0ca9d8', 'request_id': 'req-74c4826f-4024-42b5-be69-b109a1b572d6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:47.465476 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-74c4826f-4024-42b5-be69-b109a1b572d6 admin admin] The token cache is being invalidate because project cca355ef4a294ecd88cc299b7e0ca9d8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:04:47.465774 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-74c4826f-4024-42b5-be69-b109a1b572d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:47.466429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-74c4826f-4024-42b5-be69-b109a1b572d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:47.467959 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1142/2324] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:04:47 2022] DELETE /identity/v3/projects/cca355ef4a294ecd88cc299b7e0ca9d8 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:47.473632 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f5c1d7-07b8-464f-9e62-49115fbe0390 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:47.473916 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f5c1d7-07b8-464f-9e62-49115fbe0390 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:47.474324 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f5c1d7-07b8-464f-9e62-49115fbe0390 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:47.495108 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-79f5c1d7-07b8-464f-9e62-49115fbe0390 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:47.496337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-79f5c1d7-07b8-464f-9e62-49115fbe0390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:47.559212 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1183/2325] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 01:04:47.567035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711d55c5-df47-49b0-a67c-da3e60ec36f2 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:47.567423 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711d55c5-df47-49b0-a67c-da3e60ec36f2 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:47.567711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711d55c5-df47-49b0-a67c-da3e60ec36f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:47.593235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-711d55c5-df47-49b0-a67c-da3e60ec36f2 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:47.594483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-711d55c5-df47-49b0-a67c-da3e60ec36f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:47.603368 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1143/2326] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:47.607668 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f5c1d7-07b8-464f-9e62-49115fbe0390 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:47.617966 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f5c1d7-07b8-464f-9e62-49115fbe0390 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:47.622492 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247f765a-6f57-4d9b-9235-f36be61e10f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:47.623119 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247f765a-6f57-4d9b-9235-f36be61e10f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:47.623395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247f765a-6f57-4d9b-9235-f36be61e10f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:47.623664 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247f765a-6f57-4d9b-9235-f36be61e10f4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:47.624302 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247f765a-6f57-4d9b-9235-f36be61e10f4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:04:47.624604 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247f765a-6f57-4d9b-9235-f36be61e10f4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:47.626194 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247f765a-6f57-4d9b-9235-f36be61e10f4 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:47.639484 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1184/2327] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:04:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:04:47.643702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711d55c5-df47-49b0-a67c-da3e60ec36f2 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:47.653860 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711d55c5-df47-49b0-a67c-da3e60ec36f2 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:47.658469 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c99bdf-4841-4932-937a-431731b2b280 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:47.659093 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c99bdf-4841-4932-937a-431731b2b280 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:47.659373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c99bdf-4841-4932-937a-431731b2b280 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:47.659649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c99bdf-4841-4932-937a-431731b2b280 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:47.660271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c99bdf-4841-4932-937a-431731b2b280 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:47.662193 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c99bdf-4841-4932-937a-431731b2b280 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:47.689705 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1144/2328] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:04:47 2022] POST /identity/v3/projects => generated 383 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:04:47.694854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247f765a-6f57-4d9b-9235-f36be61e10f4 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:47.706417 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247f765a-6f57-4d9b-9235-f36be61e10f4 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:47.765804 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86130cd8-7d88-4f79-9c75-289fdfee7c1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:47.766342 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86130cd8-7d88-4f79-9c75-289fdfee7c1a admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:47.766627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86130cd8-7d88-4f79-9c75-289fdfee7c1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:47.766942 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86130cd8-7d88-4f79-9c75-289fdfee7c1a admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:47.767575 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86130cd8-7d88-4f79-9c75-289fdfee7c1a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:47.769557 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86130cd8-7d88-4f79-9c75-289fdfee7c1a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:47.791632 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1185/2329] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:04:47 2022] POST /identity/v3/users => generated 292 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:04:47.795891 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c99bdf-4841-4932-937a-431731b2b280 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:47.807576 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c99bdf-4841-4932-937a-431731b2b280 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:47.812274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3833ab27-79dd-4f68-bdac-9aebdc954af4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:47.812877 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3833ab27-79dd-4f68-bdac-9aebdc954af4 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:47.813161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3833ab27-79dd-4f68-bdac-9aebdc954af4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:47.813434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3833ab27-79dd-4f68-bdac-9aebdc954af4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:47.814073 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3833ab27-79dd-4f68-bdac-9aebdc954af4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:47.815740 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3833ab27-79dd-4f68-bdac-9aebdc954af4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:47.827035 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1145/2330] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:47 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:04:47.831283 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86130cd8-7d88-4f79-9c75-289fdfee7c1a admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:47.842190 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86130cd8-7d88-4f79-9c75-289fdfee7c1a admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:47.846917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71eba96-9eaa-4dbf-bb4b-42cfce20761f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:47.847548 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71eba96-9eaa-4dbf-bb4b-42cfce20761f admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:47.847840 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71eba96-9eaa-4dbf-bb4b-42cfce20761f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:47.848759 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71eba96-9eaa-4dbf-bb4b-42cfce20761f admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:47.849306 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71eba96-9eaa-4dbf-bb4b-42cfce20761f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:47.851020 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71eba96-9eaa-4dbf-bb4b-42cfce20761f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:47.853373 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:47.863354 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1186/2331] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:47 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 01:04:47.868022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3833ab27-79dd-4f68-bdac-9aebdc954af4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:47.878512 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3833ab27-79dd-4f68-bdac-9aebdc954af4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:47.883384 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170b73fc-e7ba-455e-b175-4e053ae3e1d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:47.884054 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170b73fc-e7ba-455e-b175-4e053ae3e1d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:47.884338 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170b73fc-e7ba-455e-b175-4e053ae3e1d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:47.885292 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170b73fc-e7ba-455e-b175-4e053ae3e1d1 admin admin] PATH_INFO: `/v3/projects/8e285686fd0f4ef883fbbab3a24ace5f/users/053c94205d4d497cafeb5c83c591b14e/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:47.899911 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-170b73fc-e7ba-455e-b175-4e053ae3e1d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e285686fd0f4ef883fbbab3a24ace5f, user_id=053c94205d4d497cafeb5c83c591b14e, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:47.903077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-170b73fc-e7ba-455e-b175-4e053ae3e1d1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:47.917817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1146/2332] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:04:47 2022] PUT /identity/v3/projects/8e285686fd0f4ef883fbbab3a24ace5f/users/053c94205d4d497cafeb5c83c591b14e/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:47.928905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a25a2c7-07be-4540-b1ce-d1480b4803ca None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:47.929193 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a25a2c7-07be-4540-b1ce-d1480b4803ca None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:47.929491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a25a2c7-07be-4540-b1ce-d1480b4803ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:47.960072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-9a25a2c7-07be-4540-b1ce-d1480b4803ca None None] MFA Rules not processed for user `053c94205d4d497cafeb5c83c591b14e`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:47.961307 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-9a25a2c7-07be-4540-b1ce-d1480b4803ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:48.035908 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1187/2333] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:47 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:48.047172 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170b73fc-e7ba-455e-b175-4e053ae3e1d1 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:48.048862 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-170b73fc-e7ba-455e-b175-4e053ae3e1d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:48.125321 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170b73fc-e7ba-455e-b175-4e053ae3e1d1 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:48.132478 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf40906-4812-4457-a61a-b6e5daaf077f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:48.133081 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf40906-4812-4457-a61a-b6e5daaf077f service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:48.133358 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf40906-4812-4457-a61a-b6e5daaf077f service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:48.133648 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf40906-4812-4457-a61a-b6e5daaf077f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:48.141767 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf40906-4812-4457-a61a-b6e5daaf077f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:48.143427 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf40906-4812-4457-a61a-b6e5daaf077f service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:48.152637 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1147/2334] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:48 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:50.166788 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a25a2c7-07be-4540-b1ce-d1480b4803ca None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:50.168256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-9a25a2c7-07be-4540-b1ce-d1480b4803ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:50.234667 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a25a2c7-07be-4540-b1ce-d1480b4803ca None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:50.239256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080ea243-ec6e-4786-b155-2a09c8985102 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:50.239891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080ea243-ec6e-4786-b155-2a09c8985102 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:50.240172 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080ea243-ec6e-4786-b155-2a09c8985102 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:50.240445 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080ea243-ec6e-4786-b155-2a09c8985102 admin admin] PATH_INFO: `/v3/users/3a4e6e422e77495db2523ce52ca5a13c` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:50.241562 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080ea243-ec6e-4786-b155-2a09c8985102 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3a4e6e422e77495db2523ce52ca5a13c)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:50.243614 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080ea243-ec6e-4786-b155-2a09c8985102 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:50.289741 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-080ea243-ec6e-4786-b155-2a09c8985102 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3a4e6e422e77495db2523ce52ca5a13c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:50.290395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-080ea243-ec6e-4786-b155-2a09c8985102 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3a4e6e422e77495db2523ce52ca5a13c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:50.297645 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-080ea243-ec6e-4786-b155-2a09c8985102 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3a4e6e422e77495db2523ce52ca5a13c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:50.310727 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-080ea243-ec6e-4786-b155-2a09c8985102 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3a4e6e422e77495db2523ce52ca5a13c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:50.318561 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-080ea243-ec6e-4786-b155-2a09c8985102 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3a4e6e422e77495db2523ce52ca5a13c'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:50.324868 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1188/2335] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:04:50 2022] DELETE /identity/v3/users/3a4e6e422e77495db2523ce52ca5a13c => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:50.329371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e2ba07-cbe4-4a3b-b5ac-92fb0662fb14 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:50.329905 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e2ba07-cbe4-4a3b-b5ac-92fb0662fb14 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:50.329905 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e2ba07-cbe4-4a3b-b5ac-92fb0662fb14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:50.348917 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-42e2ba07-cbe4-4a3b-b5ac-92fb0662fb14 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:50.350110 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-42e2ba07-cbe4-4a3b-b5ac-92fb0662fb14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:50.390516 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:DELETE=5,keystone:SELECT=92,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:50.411297 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1148/2336] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:50.421028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080ea243-ec6e-4786-b155-2a09c8985102 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:50.422495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-080ea243-ec6e-4786-b155-2a09c8985102 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:50.433479 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:50.483798 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080ea243-ec6e-4786-b155-2a09c8985102 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:50.487419 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4880bcf5-3b5a-43d7-a04f-afd177a79efd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:50.488018 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4880bcf5-3b5a-43d7-a04f-afd177a79efd service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:50.488296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4880bcf5-3b5a-43d7-a04f-afd177a79efd service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:50.489073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4880bcf5-3b5a-43d7-a04f-afd177a79efd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:50.498045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4880bcf5-3b5a-43d7-a04f-afd177a79efd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:50.499500 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4880bcf5-3b5a-43d7-a04f-afd177a79efd service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:50.510043 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1189/2337] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:50.774175 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e2ba07-cbe4-4a3b-b5ac-92fb0662fb14 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:50.775636 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-42e2ba07-cbe4-4a3b-b5ac-92fb0662fb14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:50.788203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e2ba07-cbe4-4a3b-b5ac-92fb0662fb14 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:50.792908 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74774b5-c97d-480d-a4e9-217a002d6c9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:50.793540 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74774b5-c97d-480d-a4e9-217a002d6c9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:50.793816 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74774b5-c97d-480d-a4e9-217a002d6c9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:50.794086 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74774b5-c97d-480d-a4e9-217a002d6c9c admin admin] PATH_INFO: `/v3/projects/290958d9e5034a9bb65cadd610860cc8` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:50.795225 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74774b5-c97d-480d-a4e9-217a002d6c9c admin admin] RBAC: Authorizing `identity:delete_project(project_id=290958d9e5034a9bb65cadd610860cc8)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:50.797144 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74774b5-c97d-480d-a4e9-217a002d6c9c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:50.845171 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c74774b5-c97d-480d-a4e9-217a002d6c9c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '290958d9e5034a9bb65cadd610860cc8', 'request_id': 'req-c74774b5-c97d-480d-a4e9-217a002d6c9c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:50.853279 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c74774b5-c97d-480d-a4e9-217a002d6c9c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '290958d9e5034a9bb65cadd610860cc8', 'request_id': 'req-c74774b5-c97d-480d-a4e9-217a002d6c9c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:50.860124 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c74774b5-c97d-480d-a4e9-217a002d6c9c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '290958d9e5034a9bb65cadd610860cc8', 'request_id': 'req-c74774b5-c97d-480d-a4e9-217a002d6c9c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:50.871379 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c74774b5-c97d-480d-a4e9-217a002d6c9c admin admin] The token cache is being invalidate because project 290958d9e5034a9bb65cadd610860cc8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:04:50.871670 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c74774b5-c97d-480d-a4e9-217a002d6c9c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:50.872402 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-c74774b5-c97d-480d-a4e9-217a002d6c9c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:50.873837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1149/2338] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:04:50 2022] DELETE /identity/v3/projects/290958d9e5034a9bb65cadd610860cc8 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:04:55.447267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4880bcf5-3b5a-43d7-a04f-afd177a79efd service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:55.447267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-4880bcf5-3b5a-43d7-a04f-afd177a79efd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:55.459470 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4880bcf5-3b5a-43d7-a04f-afd177a79efd service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:55.515591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af82d77-8f8d-41ea-97fc-df1de3eb5bb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:55.516229 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af82d77-8f8d-41ea-97fc-df1de3eb5bb9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:55.516514 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af82d77-8f8d-41ea-97fc-df1de3eb5bb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:55.516792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af82d77-8f8d-41ea-97fc-df1de3eb5bb9 admin admin] PATH_INFO: `/v3/users/053c94205d4d497cafeb5c83c591b14e` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:55.517927 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af82d77-8f8d-41ea-97fc-df1de3eb5bb9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=053c94205d4d497cafeb5c83c591b14e)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:55.519917 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af82d77-8f8d-41ea-97fc-df1de3eb5bb9 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:55.571376 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-9af82d77-8f8d-41ea-97fc-df1de3eb5bb9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '053c94205d4d497cafeb5c83c591b14e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:55.572041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-9af82d77-8f8d-41ea-97fc-df1de3eb5bb9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '053c94205d4d497cafeb5c83c591b14e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:55.579918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-9af82d77-8f8d-41ea-97fc-df1de3eb5bb9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '053c94205d4d497cafeb5c83c591b14e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:55.592735 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-9af82d77-8f8d-41ea-97fc-df1de3eb5bb9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '053c94205d4d497cafeb5c83c591b14e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:55.600203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-9af82d77-8f8d-41ea-97fc-df1de3eb5bb9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '053c94205d4d497cafeb5c83c591b14e'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:55.606116 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1190/2339] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:04:55 2022] DELETE /identity/v3/users/053c94205d4d497cafeb5c83c591b14e => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:55.610763 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2f0706-34cf-41aa-ad24-7c3159096159 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:55.611282 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2f0706-34cf-41aa-ad24-7c3159096159 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:55.611382 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2f0706-34cf-41aa-ad24-7c3159096159 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:55.630988 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-ba2f0706-34cf-41aa-ad24-7c3159096159 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:55.632186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ba2f0706-34cf-41aa-ad24-7c3159096159 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:55.691977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1150/2340] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:55.701981 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af82d77-8f8d-41ea-97fc-df1de3eb5bb9 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:55.703412 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-9af82d77-8f8d-41ea-97fc-df1de3eb5bb9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:55.737819 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:55.765354 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af82d77-8f8d-41ea-97fc-df1de3eb5bb9 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:55.769038 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1835b4-0a21-47a6-bb11-e87482ef0444 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:55.769644 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1835b4-0a21-47a6-bb11-e87482ef0444 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:55.769922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1835b4-0a21-47a6-bb11-e87482ef0444 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:55.770191 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1835b4-0a21-47a6-bb11-e87482ef0444 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:55.780093 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1835b4-0a21-47a6-bb11-e87482ef0444 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:55.781548 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1835b4-0a21-47a6-bb11-e87482ef0444 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:55.792585 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1191/2341] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:56.065797 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2f0706-34cf-41aa-ad24-7c3159096159 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:56.067351 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-ba2f0706-34cf-41aa-ad24-7c3159096159 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:56.080824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2f0706-34cf-41aa-ad24-7c3159096159 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:56.086419 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff796e23-e81c-4421-9b0d-ed4010681e07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:56.087174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff796e23-e81c-4421-9b0d-ed4010681e07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:56.087478 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff796e23-e81c-4421-9b0d-ed4010681e07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:56.087757 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff796e23-e81c-4421-9b0d-ed4010681e07 admin admin] PATH_INFO: `/v3/projects/8e285686fd0f4ef883fbbab3a24ace5f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:56.088914 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff796e23-e81c-4421-9b0d-ed4010681e07 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e285686fd0f4ef883fbbab3a24ace5f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:56.090854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff796e23-e81c-4421-9b0d-ed4010681e07 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:56.138395 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ff796e23-e81c-4421-9b0d-ed4010681e07 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e285686fd0f4ef883fbbab3a24ace5f', 'request_id': 'req-ff796e23-e81c-4421-9b0d-ed4010681e07'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:56.146047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ff796e23-e81c-4421-9b0d-ed4010681e07 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e285686fd0f4ef883fbbab3a24ace5f', 'request_id': 'req-ff796e23-e81c-4421-9b0d-ed4010681e07'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:56.153432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ff796e23-e81c-4421-9b0d-ed4010681e07 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e285686fd0f4ef883fbbab3a24ace5f', 'request_id': 'req-ff796e23-e81c-4421-9b0d-ed4010681e07'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:56.155495 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:56.166148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ff796e23-e81c-4421-9b0d-ed4010681e07 admin admin] The token cache is being invalidate because project 8e285686fd0f4ef883fbbab3a24ace5f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:04:56.166441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ff796e23-e81c-4421-9b0d-ed4010681e07 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:56.167146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-ff796e23-e81c-4421-9b0d-ed4010681e07 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:04:56.168649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1151/2342] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:04:56 2022] DELETE /identity/v3/projects/8e285686fd0f4ef883fbbab3a24ace5f => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:56.173509 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f1b11e-220e-413e-928c-135ec506bb00 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:56.173797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f1b11e-220e-413e-928c-135ec506bb00 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:56.174075 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f1b11e-220e-413e-928c-135ec506bb00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:56.194389 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-d2f1b11e-220e-413e-928c-135ec506bb00 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:56.195643 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d2f1b11e-220e-413e-928c-135ec506bb00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:56.256887 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1192/2343] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:56.265830 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a7736f-210e-4a97-b458-ca1f1dfedaec None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:56.266120 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a7736f-210e-4a97-b458-ca1f1dfedaec None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:56.266397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a7736f-210e-4a97-b458-ca1f1dfedaec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:56.287985 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-58a7736f-210e-4a97-b458-ca1f1dfedaec None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:56.289198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-58a7736f-210e-4a97-b458-ca1f1dfedaec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:56.297043 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1152/2344] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:56.301378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f1b11e-220e-413e-928c-135ec506bb00 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:56.311744 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f1b11e-220e-413e-928c-135ec506bb00 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:56.316476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a5b98a-a5a7-4905-8914-25d9dfe77607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:56.317081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a5b98a-a5a7-4905-8914-25d9dfe77607 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:56.317367 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a5b98a-a5a7-4905-8914-25d9dfe77607 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:56.317643 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a5b98a-a5a7-4905-8914-25d9dfe77607 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:56.318331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a5b98a-a5a7-4905-8914-25d9dfe77607 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:04:56.319188 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a5b98a-a5a7-4905-8914-25d9dfe77607 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:56.320803 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a5b98a-a5a7-4905-8914-25d9dfe77607 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:56.332964 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1193/2345] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:04:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 01:04:56.337257 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a7736f-210e-4a97-b458-ca1f1dfedaec None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:56.346883 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a7736f-210e-4a97-b458-ca1f1dfedaec None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:56.352105 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af95c4f-b1d4-44cf-ab67-6adee6932691 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:56.352716 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af95c4f-b1d4-44cf-ab67-6adee6932691 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:56.353004 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af95c4f-b1d4-44cf-ab67-6adee6932691 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:56.353273 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af95c4f-b1d4-44cf-ab67-6adee6932691 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:56.354284 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af95c4f-b1d4-44cf-ab67-6adee6932691 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:56.356061 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af95c4f-b1d4-44cf-ab67-6adee6932691 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:56.385538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1153/2346] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:04:56 2022] POST /identity/v3/projects => generated 371 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:04:56.389630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a5b98a-a5a7-4905-8914-25d9dfe77607 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:56.400047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a5b98a-a5a7-4905-8914-25d9dfe77607 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:56.460200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c15f968-91d6-447a-9a94-56ed07158696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:56.460831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c15f968-91d6-447a-9a94-56ed07158696 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:56.461083 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c15f968-91d6-447a-9a94-56ed07158696 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:56.461360 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c15f968-91d6-447a-9a94-56ed07158696 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:56.462005 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c15f968-91d6-447a-9a94-56ed07158696 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:56.463981 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c15f968-91d6-447a-9a94-56ed07158696 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:56.485265 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1194/2347] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:04:56 2022] POST /identity/v3/users => generated 286 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:04:56.489473 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af95c4f-b1d4-44cf-ab67-6adee6932691 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:56.500492 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af95c4f-b1d4-44cf-ab67-6adee6932691 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:56.505117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053e5c51-509d-452f-bd02-d492e676953e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:56.505717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053e5c51-509d-452f-bd02-d492e676953e admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:56.506634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053e5c51-509d-452f-bd02-d492e676953e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:56.506946 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053e5c51-509d-452f-bd02-d492e676953e admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:56.507589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053e5c51-509d-452f-bd02-d492e676953e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:56.509206 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053e5c51-509d-452f-bd02-d492e676953e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:56.519603 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1154/2348] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:56 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:04:56.523610 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c15f968-91d6-447a-9a94-56ed07158696 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:56.533594 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c15f968-91d6-447a-9a94-56ed07158696 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:56.538390 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30700631-5f71-49fb-abe3-d76c72ea24f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:56.539029 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30700631-5f71-49fb-abe3-d76c72ea24f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:56.539316 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30700631-5f71-49fb-abe3-d76c72ea24f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:56.539594 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30700631-5f71-49fb-abe3-d76c72ea24f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:56.540237 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30700631-5f71-49fb-abe3-d76c72ea24f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:56.541854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30700631-5f71-49fb-abe3-d76c72ea24f8 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:56.553397 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1195/2349] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:04:56.557464 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053e5c51-509d-452f-bd02-d492e676953e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:56.567619 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053e5c51-509d-452f-bd02-d492e676953e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:56.573213 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d43208-b768-4e16-9f2c-e70e71abe5f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:56.573864 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d43208-b768-4e16-9f2c-e70e71abe5f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:56.574152 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d43208-b768-4e16-9f2c-e70e71abe5f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:56.574425 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d43208-b768-4e16-9f2c-e70e71abe5f7 admin admin] PATH_INFO: `/v3/projects/8e94f3bfc3fb40c2abec795f711611f3/users/858a56853e2c40ac824d1bdc3b804a28/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:56.589461 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d43208-b768-4e16-9f2c-e70e71abe5f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e94f3bfc3fb40c2abec795f711611f3, user_id=858a56853e2c40ac824d1bdc3b804a28, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:56.592235 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d43208-b768-4e16-9f2c-e70e71abe5f7 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:56.605790 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1155/2350] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:04:56 2022] PUT /identity/v3/projects/8e94f3bfc3fb40c2abec795f711611f3/users/858a56853e2c40ac824d1bdc3b804a28/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:56.618769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6e7576-86aa-4c5c-8b59-f9b6fd9f4fc5 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:56.619213 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6e7576-86aa-4c5c-8b59-f9b6fd9f4fc5 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:56.619667 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6e7576-86aa-4c5c-8b59-f9b6fd9f4fc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:56.650627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-5e6e7576-86aa-4c5c-8b59-f9b6fd9f4fc5 None None] MFA Rules not processed for user `858a56853e2c40ac824d1bdc3b804a28`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:56.651798 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-5e6e7576-86aa-4c5c-8b59-f9b6fd9f4fc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:56.679959 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:56.721890 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1196/2351] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:56 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:04:56.727413 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d43208-b768-4e16-9f2c-e70e71abe5f7 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:56.789267 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d43208-b768-4e16-9f2c-e70e71abe5f7 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:56.794080 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f2e1a3-b270-46fd-9d1c-544d59ea935b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:56.794693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f2e1a3-b270-46fd-9d1c-544d59ea935b admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:56.795038 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f2e1a3-b270-46fd-9d1c-544d59ea935b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:56.795307 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f2e1a3-b270-46fd-9d1c-544d59ea935b admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:56.795961 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f2e1a3-b270-46fd-9d1c-544d59ea935b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:56.797903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f2e1a3-b270-46fd-9d1c-544d59ea935b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:56.825633 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1156/2352] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:04:56 2022] POST /identity/v3/projects => generated 373 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:04:56.829989 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6e7576-86aa-4c5c-8b59-f9b6fd9f4fc5 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:56.841271 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6e7576-86aa-4c5c-8b59-f9b6fd9f4fc5 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:56.898181 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a975d3-22c3-4303-a4b1-1ee93e278af6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:56.898808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a975d3-22c3-4303-a4b1-1ee93e278af6 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:56.899127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a975d3-22c3-4303-a4b1-1ee93e278af6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:56.899399 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a975d3-22c3-4303-a4b1-1ee93e278af6 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:56.900034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a975d3-22c3-4303-a4b1-1ee93e278af6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:56.902033 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a975d3-22c3-4303-a4b1-1ee93e278af6 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:56.923401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1197/2353] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:04:56 2022] POST /identity/v3/users => generated 293 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:04:56.927498 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f2e1a3-b270-46fd-9d1c-544d59ea935b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:56.938400 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f2e1a3-b270-46fd-9d1c-544d59ea935b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:56.943090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406e64fc-4524-454f-a0bd-b67c4404df4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:56.944034 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406e64fc-4524-454f-a0bd-b67c4404df4b admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:56.944454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406e64fc-4524-454f-a0bd-b67c4404df4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:56.944733 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406e64fc-4524-454f-a0bd-b67c4404df4b admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:56.945381 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406e64fc-4524-454f-a0bd-b67c4404df4b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:56.947287 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406e64fc-4524-454f-a0bd-b67c4404df4b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:56.960193 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1157/2354] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:56 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:04:56.964496 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a975d3-22c3-4303-a4b1-1ee93e278af6 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:56.976447 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a975d3-22c3-4303-a4b1-1ee93e278af6 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:56.981139 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c90974-9248-486a-94e1-2325a92749fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:56.981787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c90974-9248-486a-94e1-2325a92749fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:56.982064 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c90974-9248-486a-94e1-2325a92749fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:56.982332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c90974-9248-486a-94e1-2325a92749fc admin admin] PATH_INFO: `/v3/projects/47566c37dcde4c4fb5fc1998590a3819/users/866f36be9bcd4da6b2ffa5521afc6c8c/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:56.998332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c90974-9248-486a-94e1-2325a92749fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=47566c37dcde4c4fb5fc1998590a3819, user_id=866f36be9bcd4da6b2ffa5521afc6c8c, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:57.001147 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c90974-9248-486a-94e1-2325a92749fc admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:57.015493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1198/2355] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:04:56 2022] PUT /identity/v3/projects/47566c37dcde4c4fb5fc1998590a3819/users/866f36be9bcd4da6b2ffa5521afc6c8c/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:57.019765 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406e64fc-4524-454f-a0bd-b67c4404df4b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:57.030580 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406e64fc-4524-454f-a0bd-b67c4404df4b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:57.087289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8992ce40-3496-4abf-a507-811d7019cdcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:57.087899 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8992ce40-3496-4abf-a507-811d7019cdcc admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:57.088181 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8992ce40-3496-4abf-a507-811d7019cdcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:57.088456 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8992ce40-3496-4abf-a507-811d7019cdcc admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:57.089089 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8992ce40-3496-4abf-a507-811d7019cdcc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:57.090707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8992ce40-3496-4abf-a507-811d7019cdcc admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:57.092836 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:04:57.102904 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1158/2356] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:04:57 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 01:04:57.106699 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c90974-9248-486a-94e1-2325a92749fc admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:57.117392 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c90974-9248-486a-94e1-2325a92749fc admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:57.122475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56891cc-cf6b-4702-b055-c69d973ebdc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:57.123178 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56891cc-cf6b-4702-b055-c69d973ebdc1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:57.123464 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56891cc-cf6b-4702-b055-c69d973ebdc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:57.123739 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56891cc-cf6b-4702-b055-c69d973ebdc1 admin admin] PATH_INFO: `/v3/projects/47566c37dcde4c4fb5fc1998590a3819/users/866f36be9bcd4da6b2ffa5521afc6c8c/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:57.125715 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56891cc-cf6b-4702-b055-c69d973ebdc1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=47566c37dcde4c4fb5fc1998590a3819, user_id=866f36be9bcd4da6b2ffa5521afc6c8c, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:57.128436 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56891cc-cf6b-4702-b055-c69d973ebdc1 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:57.140620 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1199/2357] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:04:57 2022] PUT /identity/v3/projects/47566c37dcde4c4fb5fc1998590a3819/users/866f36be9bcd4da6b2ffa5521afc6c8c/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:04:57.151616 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3871e838-4291-42b2-89fa-36766067cf48 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:57.151907 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3871e838-4291-42b2-89fa-36766067cf48 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:57.152306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3871e838-4291-42b2-89fa-36766067cf48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:57.180863 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3871e838-4291-42b2-89fa-36766067cf48 None None] MFA Rules not processed for user `866f36be9bcd4da6b2ffa5521afc6c8c`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:04:57.182122 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3871e838-4291-42b2-89fa-36766067cf48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:57.252172 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1159/2358] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:04:57 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 01:04:57.263780 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56891cc-cf6b-4702-b055-c69d973ebdc1 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:57.265614 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b56891cc-cf6b-4702-b055-c69d973ebdc1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:04:57.324347 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56891cc-cf6b-4702-b055-c69d973ebdc1 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:57.328180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9ba637-f299-4412-b96b-afb69294e3d0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:57.328782 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9ba637-f299-4412-b96b-afb69294e3d0 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:57.329114 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9ba637-f299-4412-b96b-afb69294e3d0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:57.329331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9ba637-f299-4412-b96b-afb69294e3d0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:57.339426 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9ba637-f299-4412-b96b-afb69294e3d0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:57.340860 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9ba637-f299-4412-b96b-afb69294e3d0 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:57.351273 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1200/2359] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:57 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:04:57.427770 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3871e838-4291-42b2-89fa-36766067cf48 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:04:57.438404 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3871e838-4291-42b2-89fa-36766067cf48 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:04:57.442151 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a92cc6-7403-47fd-93f8-a1b92fbeefe0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:04:57.442758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a92cc6-7403-47fd-93f8-a1b92fbeefe0 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:04:57.443070 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a92cc6-7403-47fd-93f8-a1b92fbeefe0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:04:57.443344 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a92cc6-7403-47fd-93f8-a1b92fbeefe0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:04:57.501292 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a92cc6-7403-47fd-93f8-a1b92fbeefe0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:04:57.502730 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a92cc6-7403-47fd-93f8-a1b92fbeefe0 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:04:57.512559 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1160/2360] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:04:57 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:05:07.346092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=85,keystone:INSERT=5 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:07.508257 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=63 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:25.805169 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9ba637-f299-4412-b96b-afb69294e3d0 service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:25.806690 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-fa9ba637-f299-4412-b96b-afb69294e3d0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:25.964247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9ba637-f299-4412-b96b-afb69294e3d0 service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:25.969231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a941ad1a-034d-4e43-b932-9637f0255a94 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87785ccd37a642e08a0ad34a4fb1c754', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:25.969841 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a941ad1a-034d-4e43-b932-9637f0255a94 service placement] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:25.970122 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a941ad1a-034d-4e43-b932-9637f0255a94 service placement] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:25.970396 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a941ad1a-034d-4e43-b932-9637f0255a94 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:25.972150 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-a941ad1a-034d-4e43-b932-9637f0255a94 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:25.984430 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a941ad1a-034d-4e43-b932-9637f0255a94 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:25.986011 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a941ad1a-034d-4e43-b932-9637f0255a94 service placement] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:25.996970 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1201/2361] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:05:25 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:05:26.860198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a92cc6-7403-47fd-93f8-a1b92fbeefe0 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:26.883066 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a92cc6-7403-47fd-93f8-a1b92fbeefe0 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:26.886788 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a71ef68-b234-49d3-841d-ee4ec84728b7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:26.887612 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a71ef68-b234-49d3-841d-ee4ec84728b7 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:26.887887 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a71ef68-b234-49d3-841d-ee4ec84728b7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:26.888162 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a71ef68-b234-49d3-841d-ee4ec84728b7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:26.889934 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6a71ef68-b234-49d3-841d-ee4ec84728b7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:26.960719 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a71ef68-b234-49d3-841d-ee4ec84728b7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:26.962715 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a71ef68-b234-49d3-841d-ee4ec84728b7 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:26.973391 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1161/2362] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:05:26 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:05:28.905865 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a941ad1a-034d-4e43-b932-9637f0255a94 service placement] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:28.907360 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-a941ad1a-034d-4e43-b932-9637f0255a94 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:29.012153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:29.031537 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a941ad1a-034d-4e43-b932-9637f0255a94 service placement] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:29.037978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700f3163-2dff-43c4-9ad4-fed937566a5f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:29.039184 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700f3163-2dff-43c4-9ad4-fed937566a5f service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:29.039458 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700f3163-2dff-43c4-9ad4-fed937566a5f service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:29.039727 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700f3163-2dff-43c4-9ad4-fed937566a5f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:29.041490 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-700f3163-2dff-43c4-9ad4-fed937566a5f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:29.116238 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700f3163-2dff-43c4-9ad4-fed937566a5f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:29.118739 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700f3163-2dff-43c4-9ad4-fed937566a5f service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:29.128708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1202/2363] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:05:28 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 01:05:36.968302 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=33 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:37.328095 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842a90be-a250-4536-8e39-6abdd4bee385 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:37.328495 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842a90be-a250-4536-8e39-6abdd4bee385 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:37.328991 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842a90be-a250-4536-8e39-6abdd4bee385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:37.355554 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-842a90be-a250-4536-8e39-6abdd4bee385 None None] MFA Rules not processed for user `c4e65cef89af48a7a51e5548634de75f`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:37.358124 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-842a90be-a250-4536-8e39-6abdd4bee385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:37.366483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1162/2364] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:37 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 01:05:38.826761 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700f3163-2dff-43c4-9ad4-fed937566a5f service nova] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:38.828305 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-700f3163-2dff-43c4-9ad4-fed937566a5f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:38.922628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700f3163-2dff-43c4-9ad4-fed937566a5f service nova] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:38.926400 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b2cab9-39f4-46dd-8b65-a377c128d609 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:38.927046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b2cab9-39f4-46dd-8b65-a377c128d609 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:38.927333 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b2cab9-39f4-46dd-8b65-a377c128d609 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:38.927604 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b2cab9-39f4-46dd-8b65-a377c128d609 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:38.930344 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c1b2cab9-39f4-46dd-8b65-a377c128d609 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:39.021809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b2cab9-39f4-46dd-8b65-a377c128d609 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:39.023886 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b2cab9-39f4-46dd-8b65-a377c128d609 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:39.034344 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:39.046323 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1203/2365] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:05:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:05:40.565151 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842a90be-a250-4536-8e39-6abdd4bee385 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:40.582693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842a90be-a250-4536-8e39-6abdd4bee385 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:40.589809 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd53e3d-6bce-4f5d-a099-47b3c3f71f87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:40.591065 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd53e3d-6bce-4f5d-a099-47b3c3f71f87 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:40.591564 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd53e3d-6bce-4f5d-a099-47b3c3f71f87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:40.592050 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd53e3d-6bce-4f5d-a099-47b3c3f71f87 admin admin] PATH_INFO: `/v3/users/858a56853e2c40ac824d1bdc3b804a28` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:40.593441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd53e3d-6bce-4f5d-a099-47b3c3f71f87 admin admin] RBAC: Authorizing `identity:delete_user(user_id=858a56853e2c40ac824d1bdc3b804a28)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:40.596163 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd53e3d-6bce-4f5d-a099-47b3c3f71f87 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:40.665042 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3bd53e3d-6bce-4f5d-a099-47b3c3f71f87 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '858a56853e2c40ac824d1bdc3b804a28'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:40.665876 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3bd53e3d-6bce-4f5d-a099-47b3c3f71f87 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '858a56853e2c40ac824d1bdc3b804a28'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:40.682950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3bd53e3d-6bce-4f5d-a099-47b3c3f71f87 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '858a56853e2c40ac824d1bdc3b804a28'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:40.684227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3bd53e3d-6bce-4f5d-a099-47b3c3f71f87 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '858a56853e2c40ac824d1bdc3b804a28'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:40.699700 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3bd53e3d-6bce-4f5d-a099-47b3c3f71f87 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '858a56853e2c40ac824d1bdc3b804a28'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:40.723638 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1163/2366] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:05:40 2022] DELETE /identity/v3/users/858a56853e2c40ac824d1bdc3b804a28 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:40.725607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ea354c-ab73-4aed-9188-7370a1b641d6 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:40.726135 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ea354c-ab73-4aed-9188-7370a1b641d6 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:40.726332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ea354c-ab73-4aed-9188-7370a1b641d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:40.765294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-c6ea354c-ab73-4aed-9188-7370a1b641d6 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:40.768059 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c6ea354c-ab73-4aed-9188-7370a1b641d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:40.855190 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1204/2367] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 01:05:40.865077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd53e3d-6bce-4f5d-a099-47b3c3f71f87 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:40.867098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3bd53e3d-6bce-4f5d-a099-47b3c3f71f87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:40.957270 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd53e3d-6bce-4f5d-a099-47b3c3f71f87 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:40.961758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3484362b-766e-4ae4-94ca-389f7cbd3d9f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:40.962603 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3484362b-766e-4ae4-94ca-389f7cbd3d9f service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:40.963935 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3484362b-766e-4ae4-94ca-389f7cbd3d9f service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:40.965080 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3484362b-766e-4ae4-94ca-389f7cbd3d9f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:40.980572 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3484362b-766e-4ae4-94ca-389f7cbd3d9f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:40.982388 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3484362b-766e-4ae4-94ca-389f7cbd3d9f service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:40.998133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1164/2368] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:05:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:05:41.414460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ea354c-ab73-4aed-9188-7370a1b641d6 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:41.416050 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c6ea354c-ab73-4aed-9188-7370a1b641d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:41.435977 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ea354c-ab73-4aed-9188-7370a1b641d6 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:41.447888 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:41.448601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:41.448962 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:41.449294 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7 admin admin] PATH_INFO: `/v3/projects/8e94f3bfc3fb40c2abec795f711611f3` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:41.450632 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e94f3bfc3fb40c2abec795f711611f3)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:41.452661 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:41.516599 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e94f3bfc3fb40c2abec795f711611f3', 'request_id': 'req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:41.525774 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e94f3bfc3fb40c2abec795f711611f3', 'request_id': 'req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:41.534263 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e94f3bfc3fb40c2abec795f711611f3', 'request_id': 'req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:41.551800 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7 admin admin] The token cache is being invalidate because project 8e94f3bfc3fb40c2abec795f711611f3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:05:41.552426 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:41.554075 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:41.556403 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1205/2369] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:05:41 2022] DELETE /identity/v3/projects/8e94f3bfc3fb40c2abec795f711611f3 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:41.560306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3484362b-766e-4ae4-94ca-389f7cbd3d9f service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:41.573865 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3484362b-766e-4ae4-94ca-389f7cbd3d9f service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:41.657850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334d666c-c512-47c3-8cc5-133f2c2e48e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:41.658790 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334d666c-c512-47c3-8cc5-133f2c2e48e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:41.659497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334d666c-c512-47c3-8cc5-133f2c2e48e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:41.660243 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334d666c-c512-47c3-8cc5-133f2c2e48e0 admin admin] PATH_INFO: `/v3/users/866f36be9bcd4da6b2ffa5521afc6c8c` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:41.661757 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334d666c-c512-47c3-8cc5-133f2c2e48e0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=866f36be9bcd4da6b2ffa5521afc6c8c)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:41.665191 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334d666c-c512-47c3-8cc5-133f2c2e48e0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:41.706785 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:41.765881 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-334d666c-c512-47c3-8cc5-133f2c2e48e0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '866f36be9bcd4da6b2ffa5521afc6c8c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:41.766497 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-334d666c-c512-47c3-8cc5-133f2c2e48e0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '866f36be9bcd4da6b2ffa5521afc6c8c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:41.777324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-334d666c-c512-47c3-8cc5-133f2c2e48e0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '866f36be9bcd4da6b2ffa5521afc6c8c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:41.778141 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-334d666c-c512-47c3-8cc5-133f2c2e48e0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '866f36be9bcd4da6b2ffa5521afc6c8c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:41.804297 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-334d666c-c512-47c3-8cc5-133f2c2e48e0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '866f36be9bcd4da6b2ffa5521afc6c8c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:41.821043 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1165/2370] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:05:41 2022] DELETE /identity/v3/users/866f36be9bcd4da6b2ffa5521afc6c8c => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:42.314658 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:42.316306 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:42.334408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd2760a-43c0-478d-a6e0-1e55cd798bc7 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:42.420508 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dad39e7-66dd-445a-a9b7-8a5edadafb5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:42.421526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dad39e7-66dd-445a-a9b7-8a5edadafb5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:42.422422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dad39e7-66dd-445a-a9b7-8a5edadafb5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:42.432670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dad39e7-66dd-445a-a9b7-8a5edadafb5f admin admin] PATH_INFO: `/v3/projects/47566c37dcde4c4fb5fc1998590a3819` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:42.434508 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dad39e7-66dd-445a-a9b7-8a5edadafb5f admin admin] RBAC: Authorizing `identity:delete_project(project_id=47566c37dcde4c4fb5fc1998590a3819)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:42.436942 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dad39e7-66dd-445a-a9b7-8a5edadafb5f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:42.463640 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:42.516452 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-6dad39e7-66dd-445a-a9b7-8a5edadafb5f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '47566c37dcde4c4fb5fc1998590a3819', 'request_id': 'req-6dad39e7-66dd-445a-a9b7-8a5edadafb5f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:42.528498 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-6dad39e7-66dd-445a-a9b7-8a5edadafb5f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '47566c37dcde4c4fb5fc1998590a3819', 'request_id': 'req-6dad39e7-66dd-445a-a9b7-8a5edadafb5f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:42.547200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-6dad39e7-66dd-445a-a9b7-8a5edadafb5f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '47566c37dcde4c4fb5fc1998590a3819', 'request_id': 'req-6dad39e7-66dd-445a-a9b7-8a5edadafb5f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:42.573378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-6dad39e7-66dd-445a-a9b7-8a5edadafb5f admin admin] The token cache is being invalidate because project 47566c37dcde4c4fb5fc1998590a3819 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:05:42.574220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-6dad39e7-66dd-445a-a9b7-8a5edadafb5f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:42.575556 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-6dad39e7-66dd-445a-a9b7-8a5edadafb5f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:42.578110 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1206/2371] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:05:42 2022] DELETE /identity/v3/projects/47566c37dcde4c4fb5fc1998590a3819 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:42.582977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0549b31-bb35-4754-ad1a-934b3416750d None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:42.583495 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0549b31-bb35-4754-ad1a-934b3416750d None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:42.583615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0549b31-bb35-4754-ad1a-934b3416750d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:42.706495 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-f0549b31-bb35-4754-ad1a-934b3416750d None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:42.708337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f0549b31-bb35-4754-ad1a-934b3416750d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:42.796952 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1166/2372] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:05:42.805694 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9fccad-6be4-4868-aafc-adedbfb8d709 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:42.806796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9fccad-6be4-4868-aafc-adedbfb8d709 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:42.807952 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9fccad-6be4-4868-aafc-adedbfb8d709 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:42.836056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-7d9fccad-6be4-4868-aafc-adedbfb8d709 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:42.837946 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-7d9fccad-6be4-4868-aafc-adedbfb8d709 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:42.851756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1207/2373] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:05:42.854406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0549b31-bb35-4754-ad1a-934b3416750d None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:42.866898 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0549b31-bb35-4754-ad1a-934b3416750d None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:42.873325 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25edf36c-1f76-451d-94a0-7c122c0f0321 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:42.874508 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25edf36c-1f76-451d-94a0-7c122c0f0321 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:42.874941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25edf36c-1f76-451d-94a0-7c122c0f0321 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:42.875315 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25edf36c-1f76-451d-94a0-7c122c0f0321 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:42.876211 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25edf36c-1f76-451d-94a0-7c122c0f0321 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:05:42.876621 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25edf36c-1f76-451d-94a0-7c122c0f0321 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:42.878506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25edf36c-1f76-451d-94a0-7c122c0f0321 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:42.904339 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1167/2374] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:05:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:05:42.905886 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9fccad-6be4-4868-aafc-adedbfb8d709 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:42.923208 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9fccad-6be4-4868-aafc-adedbfb8d709 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:42.931046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd01e027-0ccb-447b-a832-79d25ffd0e75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:42.932076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd01e027-0ccb-447b-a832-79d25ffd0e75 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:42.932577 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd01e027-0ccb-447b-a832-79d25ffd0e75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:42.933101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd01e027-0ccb-447b-a832-79d25ffd0e75 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:42.935365 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd01e027-0ccb-447b-a832-79d25ffd0e75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:42.939237 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd01e027-0ccb-447b-a832-79d25ffd0e75 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:42.985177 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25edf36c-1f76-451d-94a0-7c122c0f0321 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:42.985596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1208/2375] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:05:42 2022] POST /identity/v3/projects => generated 363 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:05:42.999940 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25edf36c-1f76-451d-94a0-7c122c0f0321 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:43.090434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:43.107030 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6e0431-8cc0-4f5c-aa2e-2d841ab968ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:43.107906 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6e0431-8cc0-4f5c-aa2e-2d841ab968ec admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:43.108449 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6e0431-8cc0-4f5c-aa2e-2d841ab968ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:43.108828 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6e0431-8cc0-4f5c-aa2e-2d841ab968ec admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:43.109620 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6e0431-8cc0-4f5c-aa2e-2d841ab968ec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:43.111766 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6e0431-8cc0-4f5c-aa2e-2d841ab968ec admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:43.141124 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1168/2376] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:05:42 2022] POST /identity/v3/users => generated 282 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:05:43.146065 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd01e027-0ccb-447b-a832-79d25ffd0e75 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:43.159830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd01e027-0ccb-447b-a832-79d25ffd0e75 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:43.174168 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95b71d0-e543-4b87-bc9c-418b9061b1e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:43.174960 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95b71d0-e543-4b87-bc9c-418b9061b1e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:43.175358 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95b71d0-e543-4b87-bc9c-418b9061b1e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:43.175664 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95b71d0-e543-4b87-bc9c-418b9061b1e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:43.176465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95b71d0-e543-4b87-bc9c-418b9061b1e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:43.178856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95b71d0-e543-4b87-bc9c-418b9061b1e1 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:43.193920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1209/2377] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:05:43 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:05:43.197636 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6e0431-8cc0-4f5c-aa2e-2d841ab968ec admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:43.209103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6e0431-8cc0-4f5c-aa2e-2d841ab968ec admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:43.216098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a391917-235b-4494-8079-646a8df52b2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:43.217030 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a391917-235b-4494-8079-646a8df52b2c admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:43.217504 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a391917-235b-4494-8079-646a8df52b2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:43.217979 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a391917-235b-4494-8079-646a8df52b2c admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:43.219166 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a391917-235b-4494-8079-646a8df52b2c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:43.221012 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a391917-235b-4494-8079-646a8df52b2c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:43.237346 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1169/2378] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:05:43 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:05:43.239360 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95b71d0-e543-4b87-bc9c-418b9061b1e1 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:43.250610 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95b71d0-e543-4b87-bc9c-418b9061b1e1 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:43.255613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da866aeb-8006-436b-a9bc-636b4415aee6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:43.256314 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da866aeb-8006-436b-a9bc-636b4415aee6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:43.256666 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da866aeb-8006-436b-a9bc-636b4415aee6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:43.257092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da866aeb-8006-436b-a9bc-636b4415aee6 admin admin] PATH_INFO: `/v3/projects/87768236a9914eb7bb447042a07166ce/users/75b4ec85a8884510b2a2b22a69c84f41/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:43.275072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da866aeb-8006-436b-a9bc-636b4415aee6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=87768236a9914eb7bb447042a07166ce, user_id=75b4ec85a8884510b2a2b22a69c84f41, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:43.277781 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da866aeb-8006-436b-a9bc-636b4415aee6 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:43.310994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1210/2379] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:05:43 2022] PUT /identity/v3/projects/87768236a9914eb7bb447042a07166ce/users/75b4ec85a8884510b2a2b22a69c84f41/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:43.318930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a558e4a8-a535-440b-86b9-0d218456442c None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:43.319267 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a558e4a8-a535-440b-86b9-0d218456442c None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:43.319575 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a558e4a8-a535-440b-86b9-0d218456442c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:43.349366 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-a558e4a8-a535-440b-86b9-0d218456442c None None] MFA Rules not processed for user `75b4ec85a8884510b2a2b22a69c84f41`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:43.351384 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-a558e4a8-a535-440b-86b9-0d218456442c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:43.461702 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1170/2380] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:43 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:05:43.469199 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da866aeb-8006-436b-a9bc-636b4415aee6 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:43.557743 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da866aeb-8006-436b-a9bc-636b4415aee6 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:43.562995 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05166566-fcc2-4ed5-83b0-4ab39f3b3a15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:43.563685 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05166566-fcc2-4ed5-83b0-4ab39f3b3a15 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:43.563972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05166566-fcc2-4ed5-83b0-4ab39f3b3a15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:43.564253 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05166566-fcc2-4ed5-83b0-4ab39f3b3a15 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:43.564953 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05166566-fcc2-4ed5-83b0-4ab39f3b3a15 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:43.567657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05166566-fcc2-4ed5-83b0-4ab39f3b3a15 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:43.616880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1211/2381] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:05:43 2022] POST /identity/v3/projects => generated 363 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:05:43.618279 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a558e4a8-a535-440b-86b9-0d218456442c None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:43.630424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a558e4a8-a535-440b-86b9-0d218456442c None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:43.697998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06149312-d991-42e0-b545-44fc0fc7de0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:43.698727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06149312-d991-42e0-b545-44fc0fc7de0e admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:43.699099 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06149312-d991-42e0-b545-44fc0fc7de0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:43.699439 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06149312-d991-42e0-b545-44fc0fc7de0e admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:43.700176 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06149312-d991-42e0-b545-44fc0fc7de0e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:43.702892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06149312-d991-42e0-b545-44fc0fc7de0e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:43.731994 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1171/2382] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:05:43 2022] POST /identity/v3/users => generated 288 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:05:43.733336 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05166566-fcc2-4ed5-83b0-4ab39f3b3a15 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:43.745188 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05166566-fcc2-4ed5-83b0-4ab39f3b3a15 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:43.751257 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300f49c1-9b13-4d89-9b31-2f87226ffdd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:43.751957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300f49c1-9b13-4d89-9b31-2f87226ffdd7 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:43.752307 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300f49c1-9b13-4d89-9b31-2f87226ffdd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:43.753145 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300f49c1-9b13-4d89-9b31-2f87226ffdd7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:43.753337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300f49c1-9b13-4d89-9b31-2f87226ffdd7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:43.755104 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300f49c1-9b13-4d89-9b31-2f87226ffdd7 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:43.768909 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1212/2383] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:05:43 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:05:43.771855 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06149312-d991-42e0-b545-44fc0fc7de0e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:43.784192 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06149312-d991-42e0-b545-44fc0fc7de0e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:43.790049 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add8fa3e-a0ca-42a7-805b-7754641f3f83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:43.790889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add8fa3e-a0ca-42a7-805b-7754641f3f83 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:43.791595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add8fa3e-a0ca-42a7-805b-7754641f3f83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:43.791595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add8fa3e-a0ca-42a7-805b-7754641f3f83 admin admin] PATH_INFO: `/v3/projects/351ccc8517e14e84960b4b8b00d614ff/users/467e5995e9554aa7b9bda385eeae1a71/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:43.812078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add8fa3e-a0ca-42a7-805b-7754641f3f83 admin admin] RBAC: Authorizing `identity:create_grant(project_id=351ccc8517e14e84960b4b8b00d614ff, user_id=467e5995e9554aa7b9bda385eeae1a71, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:43.814527 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add8fa3e-a0ca-42a7-805b-7754641f3f83 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:43.831824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1172/2384] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:05:43 2022] PUT /identity/v3/projects/351ccc8517e14e84960b4b8b00d614ff/users/467e5995e9554aa7b9bda385eeae1a71/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:43.836056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300f49c1-9b13-4d89-9b31-2f87226ffdd7 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:43.847514 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300f49c1-9b13-4d89-9b31-2f87226ffdd7 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:43.857464 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:43.946645 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f8422a-655e-4e62-b83d-477dd3bade9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:43.947337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f8422a-655e-4e62-b83d-477dd3bade9e admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:43.947621 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f8422a-655e-4e62-b83d-477dd3bade9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:43.948098 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f8422a-655e-4e62-b83d-477dd3bade9e admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:43.949771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f8422a-655e-4e62-b83d-477dd3bade9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:43.952391 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f8422a-655e-4e62-b83d-477dd3bade9e admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:43.967336 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1213/2385] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:05:43 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:05:43.969496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add8fa3e-a0ca-42a7-805b-7754641f3f83 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:43.993244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add8fa3e-a0ca-42a7-805b-7754641f3f83 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:43.998403 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed26c5d-3217-47aa-916a-36cf04b0671e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:43.999133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed26c5d-3217-47aa-916a-36cf04b0671e admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:43.999409 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed26c5d-3217-47aa-916a-36cf04b0671e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:43.999681 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed26c5d-3217-47aa-916a-36cf04b0671e admin admin] PATH_INFO: `/v3/projects/351ccc8517e14e84960b4b8b00d614ff/users/467e5995e9554aa7b9bda385eeae1a71/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:44.001768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed26c5d-3217-47aa-916a-36cf04b0671e admin admin] RBAC: Authorizing `identity:create_grant(project_id=351ccc8517e14e84960b4b8b00d614ff, user_id=467e5995e9554aa7b9bda385eeae1a71, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:44.004548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed26c5d-3217-47aa-916a-36cf04b0671e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:44.024191 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1173/2386] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:05:43 2022] PUT /identity/v3/projects/351ccc8517e14e84960b4b8b00d614ff/users/467e5995e9554aa7b9bda385eeae1a71/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:44.034549 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d484ab19-3ffa-4415-81c2-ed46f9dee916 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:44.035727 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d484ab19-3ffa-4415-81c2-ed46f9dee916 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:44.036245 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d484ab19-3ffa-4415-81c2-ed46f9dee916 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:44.112525 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-d484ab19-3ffa-4415-81c2-ed46f9dee916 None None] MFA Rules not processed for user `467e5995e9554aa7b9bda385eeae1a71`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:44.115155 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d484ab19-3ffa-4415-81c2-ed46f9dee916 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:44.207721 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1214/2387] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:44 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:05:44.224582 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed26c5d-3217-47aa-916a-36cf04b0671e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:44.228338 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2ed26c5d-3217-47aa-916a-36cf04b0671e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:44.263095 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:44.380276 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed26c5d-3217-47aa-916a-36cf04b0671e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:44.386612 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2092387-d694-4829-ae6e-a71165e8fdb2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:44.387524 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2092387-d694-4829-ae6e-a71165e8fdb2 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:44.387801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2092387-d694-4829-ae6e-a71165e8fdb2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:44.388110 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2092387-d694-4829-ae6e-a71165e8fdb2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:44.401152 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2092387-d694-4829-ae6e-a71165e8fdb2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:44.402966 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2092387-d694-4829-ae6e-a71165e8fdb2 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:44.421307 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1174/2388] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:05:44 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:05:44.987176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d484ab19-3ffa-4415-81c2-ed46f9dee916 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:45.060823 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d484ab19-3ffa-4415-81c2-ed46f9dee916 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:45.065656 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c407c2fa-5b62-4a65-b43b-2934e628b599 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:45.066310 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c407c2fa-5b62-4a65-b43b-2934e628b599 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:45.066588 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c407c2fa-5b62-4a65-b43b-2934e628b599 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:45.066878 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c407c2fa-5b62-4a65-b43b-2934e628b599 admin admin] PATH_INFO: `/v3/users/75b4ec85a8884510b2a2b22a69c84f41` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:45.068035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c407c2fa-5b62-4a65-b43b-2934e628b599 admin admin] RBAC: Authorizing `identity:delete_user(user_id=75b4ec85a8884510b2a2b22a69c84f41)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:45.070052 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c407c2fa-5b62-4a65-b43b-2934e628b599 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:45.130941 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:45.155332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c407c2fa-5b62-4a65-b43b-2934e628b599 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75b4ec85a8884510b2a2b22a69c84f41'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:45.156500 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c407c2fa-5b62-4a65-b43b-2934e628b599 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75b4ec85a8884510b2a2b22a69c84f41'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:45.171444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c407c2fa-5b62-4a65-b43b-2934e628b599 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75b4ec85a8884510b2a2b22a69c84f41'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:45.203827 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c407c2fa-5b62-4a65-b43b-2934e628b599 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75b4ec85a8884510b2a2b22a69c84f41'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:45.214831 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c407c2fa-5b62-4a65-b43b-2934e628b599 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75b4ec85a8884510b2a2b22a69c84f41'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:45.229478 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1215/2389] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:05:44 2022] DELETE /identity/v3/users/75b4ec85a8884510b2a2b22a69c84f41 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:45.234601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b269472f-5167-4fda-8059-7f9faf803b84 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:45.235818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b269472f-5167-4fda-8059-7f9faf803b84 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:45.236297 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b269472f-5167-4fda-8059-7f9faf803b84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:45.263079 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b269472f-5167-4fda-8059-7f9faf803b84 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:45.267208 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b269472f-5167-4fda-8059-7f9faf803b84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:45.355215 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1175/2390] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:05:45.368623 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c407c2fa-5b62-4a65-b43b-2934e628b599 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:45.369997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c407c2fa-5b62-4a65-b43b-2934e628b599 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:45.449035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c407c2fa-5b62-4a65-b43b-2934e628b599 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:45.453411 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbcabda-7803-40ab-a78a-dcf8c129d953 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:45.454429 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbcabda-7803-40ab-a78a-dcf8c129d953 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:45.454717 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbcabda-7803-40ab-a78a-dcf8c129d953 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:45.454827 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbcabda-7803-40ab-a78a-dcf8c129d953 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:45.466182 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbcabda-7803-40ab-a78a-dcf8c129d953 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:45.467712 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbcabda-7803-40ab-a78a-dcf8c129d953 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:45.479243 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1216/2391] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:05:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:05:45.914405 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b269472f-5167-4fda-8059-7f9faf803b84 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:45.915903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b269472f-5167-4fda-8059-7f9faf803b84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:45.941551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b269472f-5167-4fda-8059-7f9faf803b84 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:45.950948 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9089f440-5d65-470f-9c10-41380c6c678c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:45.951619 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9089f440-5d65-470f-9c10-41380c6c678c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:45.951896 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9089f440-5d65-470f-9c10-41380c6c678c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:45.952173 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9089f440-5d65-470f-9c10-41380c6c678c admin admin] PATH_INFO: `/v3/projects/87768236a9914eb7bb447042a07166ce` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:45.954485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9089f440-5d65-470f-9c10-41380c6c678c admin admin] RBAC: Authorizing `identity:delete_project(project_id=87768236a9914eb7bb447042a07166ce)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:45.956489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9089f440-5d65-470f-9c10-41380c6c678c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:46.038100 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-9089f440-5d65-470f-9c10-41380c6c678c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '87768236a9914eb7bb447042a07166ce', 'request_id': 'req-9089f440-5d65-470f-9c10-41380c6c678c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:46.052494 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-9089f440-5d65-470f-9c10-41380c6c678c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '87768236a9914eb7bb447042a07166ce', 'request_id': 'req-9089f440-5d65-470f-9c10-41380c6c678c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:46.063608 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-9089f440-5d65-470f-9c10-41380c6c678c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '87768236a9914eb7bb447042a07166ce', 'request_id': 'req-9089f440-5d65-470f-9c10-41380c6c678c'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:46.090061 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-9089f440-5d65-470f-9c10-41380c6c678c admin admin] The token cache is being invalidate because project 87768236a9914eb7bb447042a07166ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:05:46.090388 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-9089f440-5d65-470f-9c10-41380c6c678c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:46.091725 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-9089f440-5d65-470f-9c10-41380c6c678c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:46.094935 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1176/2392] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:05:45 2022] DELETE /identity/v3/projects/87768236a9914eb7bb447042a07166ce => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:46.100322 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbcabda-7803-40ab-a78a-dcf8c129d953 service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:46.114849 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbcabda-7803-40ab-a78a-dcf8c129d953 service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:46.187922 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdb1b86-4b40-4b74-89e9-8fe3840b8c41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:46.188575 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdb1b86-4b40-4b74-89e9-8fe3840b8c41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:46.188869 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdb1b86-4b40-4b74-89e9-8fe3840b8c41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:46.189143 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdb1b86-4b40-4b74-89e9-8fe3840b8c41 admin admin] PATH_INFO: `/v3/users/467e5995e9554aa7b9bda385eeae1a71` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:46.190337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdb1b86-4b40-4b74-89e9-8fe3840b8c41 admin admin] RBAC: Authorizing `identity:delete_user(user_id=467e5995e9554aa7b9bda385eeae1a71)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:46.192351 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fdb1b86-4b40-4b74-89e9-8fe3840b8c41 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:46.247159 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-2fdb1b86-4b40-4b74-89e9-8fe3840b8c41 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '467e5995e9554aa7b9bda385eeae1a71'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:46.247832 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-2fdb1b86-4b40-4b74-89e9-8fe3840b8c41 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '467e5995e9554aa7b9bda385eeae1a71'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:46.256291 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-2fdb1b86-4b40-4b74-89e9-8fe3840b8c41 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '467e5995e9554aa7b9bda385eeae1a71'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:46.259373 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:46.275314 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-2fdb1b86-4b40-4b74-89e9-8fe3840b8c41 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '467e5995e9554aa7b9bda385eeae1a71'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:46.286828 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-2fdb1b86-4b40-4b74-89e9-8fe3840b8c41 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '467e5995e9554aa7b9bda385eeae1a71'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:46.294909 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1217/2393] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:05:46 2022] DELETE /identity/v3/users/467e5995e9554aa7b9bda385eeae1a71 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:46.990246 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9089f440-5d65-470f-9c10-41380c6c678c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:46.992919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-9089f440-5d65-470f-9c10-41380c6c678c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:47.018305 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9089f440-5d65-470f-9c10-41380c6c678c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:47.038588 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:47.114745 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb7b428-175c-4e77-bb08-dfaada6e567d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:47.119889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb7b428-175c-4e77-bb08-dfaada6e567d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:47.120963 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb7b428-175c-4e77-bb08-dfaada6e567d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:47.121595 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb7b428-175c-4e77-bb08-dfaada6e567d admin admin] PATH_INFO: `/v3/projects/351ccc8517e14e84960b4b8b00d614ff` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:47.123508 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb7b428-175c-4e77-bb08-dfaada6e567d admin admin] RBAC: Authorizing `identity:delete_project(project_id=351ccc8517e14e84960b4b8b00d614ff)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:47.127020 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb7b428-175c-4e77-bb08-dfaada6e567d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:47.218663 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cfb7b428-175c-4e77-bb08-dfaada6e567d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '351ccc8517e14e84960b4b8b00d614ff', 'request_id': 'req-cfb7b428-175c-4e77-bb08-dfaada6e567d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:47.232546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cfb7b428-175c-4e77-bb08-dfaada6e567d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '351ccc8517e14e84960b4b8b00d614ff', 'request_id': 'req-cfb7b428-175c-4e77-bb08-dfaada6e567d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:47.251450 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cfb7b428-175c-4e77-bb08-dfaada6e567d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '351ccc8517e14e84960b4b8b00d614ff', 'request_id': 'req-cfb7b428-175c-4e77-bb08-dfaada6e567d'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:47.263983 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cfb7b428-175c-4e77-bb08-dfaada6e567d admin admin] The token cache is being invalidate because project 351ccc8517e14e84960b4b8b00d614ff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:05:47.264305 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cfb7b428-175c-4e77-bb08-dfaada6e567d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:47.264957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-cfb7b428-175c-4e77-bb08-dfaada6e567d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:47.266514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1177/2394] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:05:46 2022] DELETE /identity/v3/projects/351ccc8517e14e84960b4b8b00d614ff => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:47.274074 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe752b5-ad31-4ff4-b9e1-14fa0ec77e92 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:47.274605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe752b5-ad31-4ff4-b9e1-14fa0ec77e92 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:47.274605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe752b5-ad31-4ff4-b9e1-14fa0ec77e92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:47.343482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-0fe752b5-ad31-4ff4-b9e1-14fa0ec77e92 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:47.347497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-0fe752b5-ad31-4ff4-b9e1-14fa0ec77e92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:47.534430 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1218/2395] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:05:47.536298 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9540113-d3e9-421c-a73d-6298ac24db68 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:47.536635 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9540113-d3e9-421c-a73d-6298ac24db68 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:47.536936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9540113-d3e9-421c-a73d-6298ac24db68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:47.561117 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b9540113-d3e9-421c-a73d-6298ac24db68 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:47.566029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b9540113-d3e9-421c-a73d-6298ac24db68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:47.576575 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1178/2396] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:05:47.580696 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe752b5-ad31-4ff4-b9e1-14fa0ec77e92 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:47.592307 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe752b5-ad31-4ff4-b9e1-14fa0ec77e92 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:47.598362 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e55299-8533-4fc6-82ed-0c8e503b4cd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:47.600232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e55299-8533-4fc6-82ed-0c8e503b4cd7 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:47.600725 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e55299-8533-4fc6-82ed-0c8e503b4cd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:47.601366 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e55299-8533-4fc6-82ed-0c8e503b4cd7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:47.602062 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e55299-8533-4fc6-82ed-0c8e503b4cd7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:05:47.602377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e55299-8533-4fc6-82ed-0c8e503b4cd7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:47.604246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e55299-8533-4fc6-82ed-0c8e503b4cd7 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:47.623606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1219/2397] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:05:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:05:47.624849 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9540113-d3e9-421c-a73d-6298ac24db68 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:47.637580 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9540113-d3e9-421c-a73d-6298ac24db68 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:47.643602 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99198c28-e217-45ce-a6a9-3928ae476fe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:47.644412 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99198c28-e217-45ce-a6a9-3928ae476fe3 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:47.645036 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99198c28-e217-45ce-a6a9-3928ae476fe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:47.645489 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99198c28-e217-45ce-a6a9-3928ae476fe3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:47.646187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99198c28-e217-45ce-a6a9-3928ae476fe3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:47.650362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99198c28-e217-45ce-a6a9-3928ae476fe3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:47.691146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1179/2398] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:05:47 2022] POST /identity/v3/projects => generated 391 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:05:47.692236 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e55299-8533-4fc6-82ed-0c8e503b4cd7 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:47.706618 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e55299-8533-4fc6-82ed-0c8e503b4cd7 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:47.814749 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62878c3a-8e8d-4f95-962c-d41eb2fe7323 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:47.815813 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62878c3a-8e8d-4f95-962c-d41eb2fe7323 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:47.815813 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62878c3a-8e8d-4f95-962c-d41eb2fe7323 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:47.816076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62878c3a-8e8d-4f95-962c-d41eb2fe7323 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:47.817175 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62878c3a-8e8d-4f95-962c-d41eb2fe7323 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:47.820300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62878c3a-8e8d-4f95-962c-d41eb2fe7323 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:47.844493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1220/2399] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:05:47 2022] POST /identity/v3/users => generated 296 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 01:05:47.848684 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99198c28-e217-45ce-a6a9-3928ae476fe3 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:47.860008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99198c28-e217-45ce-a6a9-3928ae476fe3 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:47.866537 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d9022f-4e15-44be-900e-3607cf03064e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:47.867394 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d9022f-4e15-44be-900e-3607cf03064e admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:47.867613 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d9022f-4e15-44be-900e-3607cf03064e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:47.867920 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d9022f-4e15-44be-900e-3607cf03064e admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:47.868700 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d9022f-4e15-44be-900e-3607cf03064e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:47.870443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d9022f-4e15-44be-900e-3607cf03064e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:47.889078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1180/2400] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:05:47 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:05:47.894505 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62878c3a-8e8d-4f95-962c-d41eb2fe7323 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:47.916343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62878c3a-8e8d-4f95-962c-d41eb2fe7323 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:47.922200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3dcbb3-880a-4c15-86ae-c67e4e0a8c39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:47.922987 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3dcbb3-880a-4c15-86ae-c67e4e0a8c39 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:47.923278 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3dcbb3-880a-4c15-86ae-c67e4e0a8c39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:47.923564 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3dcbb3-880a-4c15-86ae-c67e4e0a8c39 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:47.924332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3dcbb3-880a-4c15-86ae-c67e4e0a8c39 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:47.927411 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3dcbb3-880a-4c15-86ae-c67e4e0a8c39 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:47.949148 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1221/2401] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:05:47 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 01:05:47.950946 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d9022f-4e15-44be-900e-3607cf03064e admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:47.963263 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d9022f-4e15-44be-900e-3607cf03064e admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:47.969076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bba0f11-4a99-4654-91d9-b22f10d5555f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:47.969975 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bba0f11-4a99-4654-91d9-b22f10d5555f admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:47.970324 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bba0f11-4a99-4654-91d9-b22f10d5555f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:47.970619 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bba0f11-4a99-4654-91d9-b22f10d5555f admin admin] PATH_INFO: `/v3/projects/a170024b1fc84ea38cc4f49b65a114ff/users/89ca3d75e26143f9932166d506efa294/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:47.992274 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bba0f11-4a99-4654-91d9-b22f10d5555f admin admin] RBAC: Authorizing `identity:create_grant(project_id=a170024b1fc84ea38cc4f49b65a114ff, user_id=89ca3d75e26143f9932166d506efa294, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:47.995213 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bba0f11-4a99-4654-91d9-b22f10d5555f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:48.042244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1181/2402] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:05:47 2022] PUT /identity/v3/projects/a170024b1fc84ea38cc4f49b65a114ff/users/89ca3d75e26143f9932166d506efa294/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:48.065314 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313a8b29-50be-4d55-a89c-d65630d0121a None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:48.065649 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313a8b29-50be-4d55-a89c-d65630d0121a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:48.065935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313a8b29-50be-4d55-a89c-d65630d0121a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:48.098715 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-313a8b29-50be-4d55-a89c-d65630d0121a None None] MFA Rules not processed for user `89ca3d75e26143f9932166d506efa294`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:48.100633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-313a8b29-50be-4d55-a89c-d65630d0121a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:48.136928 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:48.214252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1222/2403] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:48 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:05:48.221922 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bba0f11-4a99-4654-91d9-b22f10d5555f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:48.289053 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:48.311195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bba0f11-4a99-4654-91d9-b22f10d5555f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:48.316181 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199f0ee5-2f15-4ce7-9e94-e061c78c3968 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:48.316810 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199f0ee5-2f15-4ce7-9e94-e061c78c3968 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:48.317096 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199f0ee5-2f15-4ce7-9e94-e061c78c3968 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:48.317372 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199f0ee5-2f15-4ce7-9e94-e061c78c3968 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:48.318029 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199f0ee5-2f15-4ce7-9e94-e061c78c3968 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:48.320863 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199f0ee5-2f15-4ce7-9e94-e061c78c3968 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:48.356017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1182/2404] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:05:48 2022] POST /identity/v3/projects => generated 393 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 01:05:48.359535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313a8b29-50be-4d55-a89c-d65630d0121a None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:48.377056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313a8b29-50be-4d55-a89c-d65630d0121a None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:48.473562 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22145c56-0013-41c3-a3bc-1a03a3469fe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:48.474382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22145c56-0013-41c3-a3bc-1a03a3469fe5 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:48.474382 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22145c56-0013-41c3-a3bc-1a03a3469fe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:48.474709 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22145c56-0013-41c3-a3bc-1a03a3469fe5 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:48.476024 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22145c56-0013-41c3-a3bc-1a03a3469fe5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:48.477849 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22145c56-0013-41c3-a3bc-1a03a3469fe5 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:48.508837 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1223/2405] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:05:48 2022] POST /identity/v3/users => generated 303 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:05:48.513210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199f0ee5-2f15-4ce7-9e94-e061c78c3968 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:48.524954 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199f0ee5-2f15-4ce7-9e94-e061c78c3968 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:48.531189 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70457bdd-741a-460e-a087-46eda683f4a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:48.532256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70457bdd-741a-460e-a087-46eda683f4a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:48.532552 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70457bdd-741a-460e-a087-46eda683f4a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:48.532808 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70457bdd-741a-460e-a087-46eda683f4a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:48.534278 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70457bdd-741a-460e-a087-46eda683f4a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:48.535593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70457bdd-741a-460e-a087-46eda683f4a1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:48.556247 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1183/2406] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:05:48 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:05:48.562167 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22145c56-0013-41c3-a3bc-1a03a3469fe5 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:48.577330 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22145c56-0013-41c3-a3bc-1a03a3469fe5 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:48.583555 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b648b0f6-18fa-4a3d-a31c-0f0e3640c13d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:48.587318 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b648b0f6-18fa-4a3d-a31c-0f0e3640c13d admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:48.588570 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b648b0f6-18fa-4a3d-a31c-0f0e3640c13d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:48.590279 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b648b0f6-18fa-4a3d-a31c-0f0e3640c13d admin admin] PATH_INFO: `/v3/projects/ea199a99cffe493ca2e2ec31d20ca500/users/0a105b54df80480f95ebcbef194dd409/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:48.620780 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b648b0f6-18fa-4a3d-a31c-0f0e3640c13d admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea199a99cffe493ca2e2ec31d20ca500, user_id=0a105b54df80480f95ebcbef194dd409, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:48.626703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b648b0f6-18fa-4a3d-a31c-0f0e3640c13d admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:48.651749 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1224/2407] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:05:48 2022] PUT /identity/v3/projects/ea199a99cffe493ca2e2ec31d20ca500/users/0a105b54df80480f95ebcbef194dd409/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:48.655956 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70457bdd-741a-460e-a087-46eda683f4a1 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:48.670459 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70457bdd-741a-460e-a087-46eda683f4a1 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:48.749752 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b80692-bcb5-4eb9-b253-6cfa25c02d77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:48.750384 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b80692-bcb5-4eb9-b253-6cfa25c02d77 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:48.750679 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b80692-bcb5-4eb9-b253-6cfa25c02d77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:48.751010 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b80692-bcb5-4eb9-b253-6cfa25c02d77 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:48.751662 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b80692-bcb5-4eb9-b253-6cfa25c02d77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:48.753704 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b80692-bcb5-4eb9-b253-6cfa25c02d77 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:48.765921 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1184/2408] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:05:48 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:05:48.770960 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b648b0f6-18fa-4a3d-a31c-0f0e3640c13d admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:48.783702 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b648b0f6-18fa-4a3d-a31c-0f0e3640c13d admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:48.790357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ca9b1d-6204-41e7-80c8-254a279678f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:48.790915 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ca9b1d-6204-41e7-80c8-254a279678f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:48.791217 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ca9b1d-6204-41e7-80c8-254a279678f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:48.791495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ca9b1d-6204-41e7-80c8-254a279678f9 admin admin] PATH_INFO: `/v3/projects/ea199a99cffe493ca2e2ec31d20ca500/users/0a105b54df80480f95ebcbef194dd409/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:48.793863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ca9b1d-6204-41e7-80c8-254a279678f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea199a99cffe493ca2e2ec31d20ca500, user_id=0a105b54df80480f95ebcbef194dd409, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:48.796671 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ca9b1d-6204-41e7-80c8-254a279678f9 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:48.811178 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1225/2409] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:05:48 2022] PUT /identity/v3/projects/ea199a99cffe493ca2e2ec31d20ca500/users/0a105b54df80480f95ebcbef194dd409/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:05:48.823480 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45dc6ede-8e3f-4fa7-8cdb-8de593d5c244 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:48.823923 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45dc6ede-8e3f-4fa7-8cdb-8de593d5c244 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:48.824079 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45dc6ede-8e3f-4fa7-8cdb-8de593d5c244 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:48.883724 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-45dc6ede-8e3f-4fa7-8cdb-8de593d5c244 None None] MFA Rules not processed for user `0a105b54df80480f95ebcbef194dd409`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:48.885653 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-45dc6ede-8e3f-4fa7-8cdb-8de593d5c244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:49.000927 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1185/2410] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:48 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:05:49.018313 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ca9b1d-6204-41e7-80c8-254a279678f9 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:49.020262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-12ca9b1d-6204-41e7-80c8-254a279678f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:49.135017 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ca9b1d-6204-41e7-80c8-254a279678f9 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:49.149493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158687c2-b6b0-487e-8cf1-e79dcb559475 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:49.150609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158687c2-b6b0-487e-8cf1-e79dcb559475 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:49.150609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158687c2-b6b0-487e-8cf1-e79dcb559475 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:49.150826 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158687c2-b6b0-487e-8cf1-e79dcb559475 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:49.162832 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158687c2-b6b0-487e-8cf1-e79dcb559475 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:49.164344 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158687c2-b6b0-487e-8cf1-e79dcb559475 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:49.175645 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1226/2411] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:05:49 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:05:49.250516 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45dc6ede-8e3f-4fa7-8cdb-8de593d5c244 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:49.252103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-45dc6ede-8e3f-4fa7-8cdb-8de593d5c244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:49.278757 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45dc6ede-8e3f-4fa7-8cdb-8de593d5c244 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:49.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec6ece8-07ca-460d-ae7b-1071de98587a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:49.289424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec6ece8-07ca-460d-ae7b-1071de98587a service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:49.289709 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec6ece8-07ca-460d-ae7b-1071de98587a service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:49.289984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec6ece8-07ca-460d-ae7b-1071de98587a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:49.346199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:49.382228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec6ece8-07ca-460d-ae7b-1071de98587a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:49.384277 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec6ece8-07ca-460d-ae7b-1071de98587a service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:49.396000 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1186/2412] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:05:49 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:05:49.840985 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158687c2-b6b0-487e-8cf1-e79dcb559475 service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:49.842617 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-158687c2-b6b0-487e-8cf1-e79dcb559475 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:49.880560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:49.950744 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158687c2-b6b0-487e-8cf1-e79dcb559475 service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:49.956797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71726d65-fbc3-44e0-a165-5619716707fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:49.957781 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71726d65-fbc3-44e0-a165-5619716707fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:49.958158 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71726d65-fbc3-44e0-a165-5619716707fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:49.959153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71726d65-fbc3-44e0-a165-5619716707fd admin admin] PATH_INFO: `/v3/users/b899e194369a4efcb87d7cb92dc5f902` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:49.961048 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71726d65-fbc3-44e0-a165-5619716707fd admin admin] RBAC: Authorizing `identity:delete_user(user_id=b899e194369a4efcb87d7cb92dc5f902)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:49.963103 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71726d65-fbc3-44e0-a165-5619716707fd admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:50.034430 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-71726d65-fbc3-44e0-a165-5619716707fd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b899e194369a4efcb87d7cb92dc5f902'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:50.035157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-71726d65-fbc3-44e0-a165-5619716707fd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b899e194369a4efcb87d7cb92dc5f902'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:50.045913 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-71726d65-fbc3-44e0-a165-5619716707fd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b899e194369a4efcb87d7cb92dc5f902'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:50.059974 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-71726d65-fbc3-44e0-a165-5619716707fd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b899e194369a4efcb87d7cb92dc5f902'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:50.067481 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-71726d65-fbc3-44e0-a165-5619716707fd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b899e194369a4efcb87d7cb92dc5f902'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:50.074699 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1227/2413] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:05:49 2022] DELETE /identity/v3/users/b899e194369a4efcb87d7cb92dc5f902 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:50.263600 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec6ece8-07ca-460d-ae7b-1071de98587a service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:50.282591 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec6ece8-07ca-460d-ae7b-1071de98587a service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:50.348047 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a79255-c425-4307-b258-0a7d5b247547 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:50.348693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a79255-c425-4307-b258-0a7d5b247547 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:50.348977 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a79255-c425-4307-b258-0a7d5b247547 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:50.349252 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a79255-c425-4307-b258-0a7d5b247547 admin admin] PATH_INFO: `/v3/projects/ed24cab62e1f4810a10e7ba36fcbeadb` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:50.350425 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a79255-c425-4307-b258-0a7d5b247547 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ed24cab62e1f4810a10e7ba36fcbeadb)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:50.352399 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a79255-c425-4307-b258-0a7d5b247547 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:50.411337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-78a79255-c425-4307-b258-0a7d5b247547 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ed24cab62e1f4810a10e7ba36fcbeadb', 'request_id': 'req-78a79255-c425-4307-b258-0a7d5b247547'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:50.419950 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-78a79255-c425-4307-b258-0a7d5b247547 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ed24cab62e1f4810a10e7ba36fcbeadb', 'request_id': 'req-78a79255-c425-4307-b258-0a7d5b247547'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:50.429109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-78a79255-c425-4307-b258-0a7d5b247547 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ed24cab62e1f4810a10e7ba36fcbeadb', 'request_id': 'req-78a79255-c425-4307-b258-0a7d5b247547'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:50.440484 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-78a79255-c425-4307-b258-0a7d5b247547 admin admin] The token cache is being invalidate because project ed24cab62e1f4810a10e7ba36fcbeadb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:05:50.440773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-78a79255-c425-4307-b258-0a7d5b247547 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:50.441547 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-78a79255-c425-4307-b258-0a7d5b247547 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:50.442981 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1187/2414] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:05:50 2022] DELETE /identity/v3/projects/ed24cab62e1f4810a10e7ba36fcbeadb => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:50.450304 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71726d65-fbc3-44e0-a165-5619716707fd admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:50.451854 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-71726d65-fbc3-44e0-a165-5619716707fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:50.466408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71726d65-fbc3-44e0-a165-5619716707fd admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:50.530071 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41361992-5f0a-4d6b-806e-4d09cab41999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:50.530730 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41361992-5f0a-4d6b-806e-4d09cab41999 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:50.531046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41361992-5f0a-4d6b-806e-4d09cab41999 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:50.531324 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41361992-5f0a-4d6b-806e-4d09cab41999 admin admin] PATH_INFO: `/v3/users/847949f2b2f84783bcad2fcd9177e811` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:50.532538 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41361992-5f0a-4d6b-806e-4d09cab41999 admin admin] RBAC: Authorizing `identity:delete_user(user_id=847949f2b2f84783bcad2fcd9177e811)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:50.534481 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41361992-5f0a-4d6b-806e-4d09cab41999 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:50.591841 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:50.593642 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-41361992-5f0a-4d6b-806e-4d09cab41999 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '847949f2b2f84783bcad2fcd9177e811'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:50.594348 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-41361992-5f0a-4d6b-806e-4d09cab41999 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '847949f2b2f84783bcad2fcd9177e811'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:50.604771 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-41361992-5f0a-4d6b-806e-4d09cab41999 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '847949f2b2f84783bcad2fcd9177e811'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:50.622267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-41361992-5f0a-4d6b-806e-4d09cab41999 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '847949f2b2f84783bcad2fcd9177e811'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:50.631049 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-41361992-5f0a-4d6b-806e-4d09cab41999 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '847949f2b2f84783bcad2fcd9177e811'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:50.638115 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1228/2415] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:05:50 2022] DELETE /identity/v3/users/847949f2b2f84783bcad2fcd9177e811 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:50.830397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a79255-c425-4307-b258-0a7d5b247547 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:50.832187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-78a79255-c425-4307-b258-0a7d5b247547 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:50.848872 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a79255-c425-4307-b258-0a7d5b247547 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:50.914729 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ac5000-dee0-43ed-be10-d564ef577ca3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:50.915706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ac5000-dee0-43ed-be10-d564ef577ca3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:50.915991 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ac5000-dee0-43ed-be10-d564ef577ca3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:50.916495 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ac5000-dee0-43ed-be10-d564ef577ca3 admin admin] PATH_INFO: `/v3/projects/ef2df25edb2b4bee9f71c94e2249e1f7` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:50.917500 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ac5000-dee0-43ed-be10-d564ef577ca3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ef2df25edb2b4bee9f71c94e2249e1f7)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:50.919508 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ac5000-dee0-43ed-be10-d564ef577ca3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:50.932664 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:50.984301 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-69ac5000-dee0-43ed-be10-d564ef577ca3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ef2df25edb2b4bee9f71c94e2249e1f7', 'request_id': 'req-69ac5000-dee0-43ed-be10-d564ef577ca3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:50.993547 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-69ac5000-dee0-43ed-be10-d564ef577ca3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ef2df25edb2b4bee9f71c94e2249e1f7', 'request_id': 'req-69ac5000-dee0-43ed-be10-d564ef577ca3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:51.007297 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-69ac5000-dee0-43ed-be10-d564ef577ca3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ef2df25edb2b4bee9f71c94e2249e1f7', 'request_id': 'req-69ac5000-dee0-43ed-be10-d564ef577ca3'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:51.021690 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-69ac5000-dee0-43ed-be10-d564ef577ca3 admin admin] The token cache is being invalidate because project ef2df25edb2b4bee9f71c94e2249e1f7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:05:51.022277 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-69ac5000-dee0-43ed-be10-d564ef577ca3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:51.023182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-69ac5000-dee0-43ed-be10-d564ef577ca3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:51.027887 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1188/2416] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:05:50 2022] DELETE /identity/v3/projects/ef2df25edb2b4bee9f71c94e2249e1f7 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:53.104230 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41361992-5f0a-4d6b-806e-4d09cab41999 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:53.105663 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-41361992-5f0a-4d6b-806e-4d09cab41999 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:53.118312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41361992-5f0a-4d6b-806e-4d09cab41999 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:53.174916 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c254a420-dda9-41d4-982b-e2969053b55c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:53.175547 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c254a420-dda9-41d4-982b-e2969053b55c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:53.175827 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c254a420-dda9-41d4-982b-e2969053b55c admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:53.176098 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c254a420-dda9-41d4-982b-e2969053b55c admin admin] PATH_INFO: `/v3/users/89ca3d75e26143f9932166d506efa294` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:53.177198 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c254a420-dda9-41d4-982b-e2969053b55c admin admin] RBAC: Authorizing `identity:delete_user(user_id=89ca3d75e26143f9932166d506efa294)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:53.179656 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c254a420-dda9-41d4-982b-e2969053b55c admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:53.230171 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c254a420-dda9-41d4-982b-e2969053b55c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89ca3d75e26143f9932166d506efa294'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:53.230748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c254a420-dda9-41d4-982b-e2969053b55c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89ca3d75e26143f9932166d506efa294'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:53.238111 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c254a420-dda9-41d4-982b-e2969053b55c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89ca3d75e26143f9932166d506efa294'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:53.252262 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c254a420-dda9-41d4-982b-e2969053b55c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89ca3d75e26143f9932166d506efa294'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:53.259278 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c254a420-dda9-41d4-982b-e2969053b55c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89ca3d75e26143f9932166d506efa294'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:53.265416 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1229/2417] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:05:53 2022] DELETE /identity/v3/users/89ca3d75e26143f9932166d506efa294 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:53.269885 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3281e0-a11a-4b33-b64a-27088a8e7788 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:53.270406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3281e0-a11a-4b33-b64a-27088a8e7788 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:53.270406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3281e0-a11a-4b33-b64a-27088a8e7788 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:53.289926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-3d3281e0-a11a-4b33-b64a-27088a8e7788 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:53.291185 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3d3281e0-a11a-4b33-b64a-27088a8e7788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:53.350392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1189/2418] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:05:53.360085 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c254a420-dda9-41d4-982b-e2969053b55c admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:53.361526 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c254a420-dda9-41d4-982b-e2969053b55c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:53.416268 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=85,keystone:INSERT=2,keystone:DELETE=13 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:53.430957 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c254a420-dda9-41d4-982b-e2969053b55c admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:53.436498 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f986e444-0188-4f41-b99a-31bd0483c735 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:53.437169 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f986e444-0188-4f41-b99a-31bd0483c735 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:53.437698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f986e444-0188-4f41-b99a-31bd0483c735 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:53.438061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f986e444-0188-4f41-b99a-31bd0483c735 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:53.448727 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f986e444-0188-4f41-b99a-31bd0483c735 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:53.450180 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f986e444-0188-4f41-b99a-31bd0483c735 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:53.460281 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1230/2419] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:05:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:05:53.764987 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3281e0-a11a-4b33-b64a-27088a8e7788 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:53.766466 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3d3281e0-a11a-4b33-b64a-27088a8e7788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:53.781634 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3281e0-a11a-4b33-b64a-27088a8e7788 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:53.788138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afdc2d6-aed2-485d-a5ec-365488a30959 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:53.789057 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afdc2d6-aed2-485d-a5ec-365488a30959 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:53.789350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afdc2d6-aed2-485d-a5ec-365488a30959 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:53.789645 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afdc2d6-aed2-485d-a5ec-365488a30959 admin admin] PATH_INFO: `/v3/projects/a170024b1fc84ea38cc4f49b65a114ff` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:53.790984 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afdc2d6-aed2-485d-a5ec-365488a30959 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a170024b1fc84ea38cc4f49b65a114ff)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:53.793079 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afdc2d6-aed2-485d-a5ec-365488a30959 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:53.844283 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6afdc2d6-aed2-485d-a5ec-365488a30959 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a170024b1fc84ea38cc4f49b65a114ff', 'request_id': 'req-6afdc2d6-aed2-485d-a5ec-365488a30959'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:53.852736 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6afdc2d6-aed2-485d-a5ec-365488a30959 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a170024b1fc84ea38cc4f49b65a114ff', 'request_id': 'req-6afdc2d6-aed2-485d-a5ec-365488a30959'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:53.859699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6afdc2d6-aed2-485d-a5ec-365488a30959 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a170024b1fc84ea38cc4f49b65a114ff', 'request_id': 'req-6afdc2d6-aed2-485d-a5ec-365488a30959'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:53.872205 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6afdc2d6-aed2-485d-a5ec-365488a30959 admin admin] The token cache is being invalidate because project a170024b1fc84ea38cc4f49b65a114ff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:05:53.872504 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6afdc2d6-aed2-485d-a5ec-365488a30959 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:53.873169 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-6afdc2d6-aed2-485d-a5ec-365488a30959 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:53.880217 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1190/2420] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:05:53 2022] DELETE /identity/v3/projects/a170024b1fc84ea38cc4f49b65a114ff => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:53.884554 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f986e444-0188-4f41-b99a-31bd0483c735 service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:53.894731 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f986e444-0188-4f41-b99a-31bd0483c735 service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:53.956669 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27be69d5-9a24-4ac7-93eb-f91fd1180bd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:53.957418 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27be69d5-9a24-4ac7-93eb-f91fd1180bd4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:53.957631 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27be69d5-9a24-4ac7-93eb-f91fd1180bd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:53.957908 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27be69d5-9a24-4ac7-93eb-f91fd1180bd4 admin admin] PATH_INFO: `/v3/users/0a105b54df80480f95ebcbef194dd409` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:53.959596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27be69d5-9a24-4ac7-93eb-f91fd1180bd4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0a105b54df80480f95ebcbef194dd409)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:53.962000 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27be69d5-9a24-4ac7-93eb-f91fd1180bd4 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:54.015381 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-27be69d5-9a24-4ac7-93eb-f91fd1180bd4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0a105b54df80480f95ebcbef194dd409'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:54.016084 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-27be69d5-9a24-4ac7-93eb-f91fd1180bd4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0a105b54df80480f95ebcbef194dd409'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:54.025397 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-27be69d5-9a24-4ac7-93eb-f91fd1180bd4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0a105b54df80480f95ebcbef194dd409'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:54.039173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-27be69d5-9a24-4ac7-93eb-f91fd1180bd4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0a105b54df80480f95ebcbef194dd409'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:54.046298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-27be69d5-9a24-4ac7-93eb-f91fd1180bd4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0a105b54df80480f95ebcbef194dd409'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:54.052811 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1231/2421] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:05:53 2022] DELETE /identity/v3/users/0a105b54df80480f95ebcbef194dd409 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:54.353397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afdc2d6-aed2-485d-a5ec-365488a30959 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:54.354852 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-6afdc2d6-aed2-485d-a5ec-365488a30959 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:54.370596 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afdc2d6-aed2-485d-a5ec-365488a30959 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:54.391528 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:54.449700 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606d0aab-8997-44dc-b94e-79b28bfbd284 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:54.450349 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606d0aab-8997-44dc-b94e-79b28bfbd284 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:54.450636 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606d0aab-8997-44dc-b94e-79b28bfbd284 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:54.450953 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606d0aab-8997-44dc-b94e-79b28bfbd284 admin admin] PATH_INFO: `/v3/projects/ea199a99cffe493ca2e2ec31d20ca500` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:54.452210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606d0aab-8997-44dc-b94e-79b28bfbd284 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea199a99cffe493ca2e2ec31d20ca500)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:54.454165 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606d0aab-8997-44dc-b94e-79b28bfbd284 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:54.506140 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-606d0aab-8997-44dc-b94e-79b28bfbd284 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea199a99cffe493ca2e2ec31d20ca500', 'request_id': 'req-606d0aab-8997-44dc-b94e-79b28bfbd284'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:54.515256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-606d0aab-8997-44dc-b94e-79b28bfbd284 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea199a99cffe493ca2e2ec31d20ca500', 'request_id': 'req-606d0aab-8997-44dc-b94e-79b28bfbd284'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:54.523370 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-606d0aab-8997-44dc-b94e-79b28bfbd284 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea199a99cffe493ca2e2ec31d20ca500', 'request_id': 'req-606d0aab-8997-44dc-b94e-79b28bfbd284'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:54.536346 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-606d0aab-8997-44dc-b94e-79b28bfbd284 admin admin] The token cache is being invalidate because project ea199a99cffe493ca2e2ec31d20ca500 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:05:54.536644 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-606d0aab-8997-44dc-b94e-79b28bfbd284 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:54.537752 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-606d0aab-8997-44dc-b94e-79b28bfbd284 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:54.539329 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1191/2422] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:05:54 2022] DELETE /identity/v3/projects/ea199a99cffe493ca2e2ec31d20ca500 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:54.544367 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd810e2-ca67-44ae-ac98-1416bcb9d079 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:54.544913 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd810e2-ca67-44ae-ac98-1416bcb9d079 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:54.544913 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd810e2-ca67-44ae-ac98-1416bcb9d079 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:54.565878 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-ffd810e2-ca67-44ae-ac98-1416bcb9d079 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:54.567152 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-ffd810e2-ca67-44ae-ac98-1416bcb9d079 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:54.648656 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1232/2423] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 01:05:54.657092 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90736432-229e-4270-a3bf-908e7119c732 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:54.657378 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90736432-229e-4270-a3bf-908e7119c732 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:54.657654 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90736432-229e-4270-a3bf-908e7119c732 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:54.679610 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-90736432-229e-4270-a3bf-908e7119c732 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:54.680842 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-90736432-229e-4270-a3bf-908e7119c732 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:54.688834 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1192/2424] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:05:54.693815 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd810e2-ca67-44ae-ac98-1416bcb9d079 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:54.700423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:54.704772 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd810e2-ca67-44ae-ac98-1416bcb9d079 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:54.711059 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf475d3-f13b-40d2-bc6d-1fbd941e657b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:54.711847 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf475d3-f13b-40d2-bc6d-1fbd941e657b admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:54.712142 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf475d3-f13b-40d2-bc6d-1fbd941e657b admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:54.712427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf475d3-f13b-40d2-bc6d-1fbd941e657b admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:54.713257 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf475d3-f13b-40d2-bc6d-1fbd941e657b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:05:54.713751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf475d3-f13b-40d2-bc6d-1fbd941e657b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:54.715427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf475d3-f13b-40d2-bc6d-1fbd941e657b admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:54.732777 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1233/2425] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:05:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:05:54.737312 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90736432-229e-4270-a3bf-908e7119c732 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:54.747757 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90736432-229e-4270-a3bf-908e7119c732 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:54.752698 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872bbfdb-5a03-4d34-9def-b80435c8fb3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:54.753306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872bbfdb-5a03-4d34-9def-b80435c8fb3f admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:54.753581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872bbfdb-5a03-4d34-9def-b80435c8fb3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:54.753848 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872bbfdb-5a03-4d34-9def-b80435c8fb3f admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:54.754485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872bbfdb-5a03-4d34-9def-b80435c8fb3f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:54.756978 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872bbfdb-5a03-4d34-9def-b80435c8fb3f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:54.785872 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1193/2426] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:05:54 2022] POST /identity/v3/projects => generated 385 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:05:54.790239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf475d3-f13b-40d2-bc6d-1fbd941e657b admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:54.800573 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf475d3-f13b-40d2-bc6d-1fbd941e657b admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:54.862030 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b685b809-3f20-4007-a66e-1ed8c31a14cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:54.862757 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b685b809-3f20-4007-a66e-1ed8c31a14cb admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:54.863117 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b685b809-3f20-4007-a66e-1ed8c31a14cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:54.863117 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b685b809-3f20-4007-a66e-1ed8c31a14cb admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:54.863750 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b685b809-3f20-4007-a66e-1ed8c31a14cb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:54.866065 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b685b809-3f20-4007-a66e-1ed8c31a14cb admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:54.888484 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1234/2427] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:05:54 2022] POST /identity/v3/users => generated 293 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:05:54.892799 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872bbfdb-5a03-4d34-9def-b80435c8fb3f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:54.903756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872bbfdb-5a03-4d34-9def-b80435c8fb3f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:54.908523 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed11563c-838a-436c-a3b5-8b24c53d0ec8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:54.909234 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed11563c-838a-436c-a3b5-8b24c53d0ec8 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:54.909617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed11563c-838a-436c-a3b5-8b24c53d0ec8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:54.910007 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed11563c-838a-436c-a3b5-8b24c53d0ec8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:54.910906 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed11563c-838a-436c-a3b5-8b24c53d0ec8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:54.913297 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed11563c-838a-436c-a3b5-8b24c53d0ec8 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:54.923862 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1194/2428] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:05:54 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:05:54.928278 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b685b809-3f20-4007-a66e-1ed8c31a14cb admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:54.938447 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b685b809-3f20-4007-a66e-1ed8c31a14cb admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:54.943121 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744e215a-f42d-487c-9bf9-999d4da06bbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:54.943718 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744e215a-f42d-487c-9bf9-999d4da06bbd admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:54.944001 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744e215a-f42d-487c-9bf9-999d4da06bbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:54.944337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744e215a-f42d-487c-9bf9-999d4da06bbd admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:54.944918 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744e215a-f42d-487c-9bf9-999d4da06bbd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:54.947023 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744e215a-f42d-487c-9bf9-999d4da06bbd admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:54.957747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1235/2429] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:05:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:05:54.962447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed11563c-838a-436c-a3b5-8b24c53d0ec8 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:54.972550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed11563c-838a-436c-a3b5-8b24c53d0ec8 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:54.978659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a08564-faca-49c3-aa64-4fa33b87ea20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:54.979443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a08564-faca-49c3-aa64-4fa33b87ea20 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:54.979723 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a08564-faca-49c3-aa64-4fa33b87ea20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:54.979995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a08564-faca-49c3-aa64-4fa33b87ea20 admin admin] PATH_INFO: `/v3/projects/5ea962541a1e43c9a6df32518a591d45/users/9db29765b5224a919e2143d82f710713/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:54.995707 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a08564-faca-49c3-aa64-4fa33b87ea20 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ea962541a1e43c9a6df32518a591d45, user_id=9db29765b5224a919e2143d82f710713, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:54.998441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a08564-faca-49c3-aa64-4fa33b87ea20 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:55.013929 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1195/2430] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:05:54 2022] PUT /identity/v3/projects/5ea962541a1e43c9a6df32518a591d45/users/9db29765b5224a919e2143d82f710713/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:55.024832 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38098e2-9104-4272-9ed4-8f80f8d5a31f None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:55.025124 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38098e2-9104-4272-9ed4-8f80f8d5a31f None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:55.025404 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38098e2-9104-4272-9ed4-8f80f8d5a31f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:55.060604 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-d38098e2-9104-4272-9ed4-8f80f8d5a31f None None] MFA Rules not processed for user `9db29765b5224a919e2143d82f710713`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:55.061842 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d38098e2-9104-4272-9ed4-8f80f8d5a31f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:55.126935 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1236/2431] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:55 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:05:55.132066 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a08564-faca-49c3-aa64-4fa33b87ea20 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:55.185090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:55.201103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a08564-faca-49c3-aa64-4fa33b87ea20 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:55.205897 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b27595e-257f-4d17-94be-0ea8d524744b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:55.206510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b27595e-257f-4d17-94be-0ea8d524744b admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:55.211529 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b27595e-257f-4d17-94be-0ea8d524744b admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:55.211529 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b27595e-257f-4d17-94be-0ea8d524744b admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:55.211529 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b27595e-257f-4d17-94be-0ea8d524744b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:55.211529 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b27595e-257f-4d17-94be-0ea8d524744b admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:55.238146 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1196/2432] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:05:55 2022] POST /identity/v3/projects => generated 385 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:05:55.242427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38098e2-9104-4272-9ed4-8f80f8d5a31f None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:55.253334 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38098e2-9104-4272-9ed4-8f80f8d5a31f None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:55.310576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cef9c64-ece5-4ff6-be24-fa4c22186b9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:55.311232 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cef9c64-ece5-4ff6-be24-fa4c22186b9d admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:55.311521 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cef9c64-ece5-4ff6-be24-fa4c22186b9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:55.311800 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cef9c64-ece5-4ff6-be24-fa4c22186b9d admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:55.312442 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cef9c64-ece5-4ff6-be24-fa4c22186b9d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:55.314383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cef9c64-ece5-4ff6-be24-fa4c22186b9d admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:55.323196 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:55.338397 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1237/2433] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:05:55 2022] POST /identity/v3/users => generated 299 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:05:55.342751 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b27595e-257f-4d17-94be-0ea8d524744b admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:55.353766 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b27595e-257f-4d17-94be-0ea8d524744b admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:55.358421 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277e4a27-fc83-44f6-8b2c-e49a577cca79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:55.359059 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277e4a27-fc83-44f6-8b2c-e49a577cca79 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:55.359346 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277e4a27-fc83-44f6-8b2c-e49a577cca79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:55.359622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277e4a27-fc83-44f6-8b2c-e49a577cca79 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:55.360302 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277e4a27-fc83-44f6-8b2c-e49a577cca79 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:55.361906 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277e4a27-fc83-44f6-8b2c-e49a577cca79 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:55.373385 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1197/2434] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:05:55 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 01:05:55.377640 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cef9c64-ece5-4ff6-be24-fa4c22186b9d admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:55.387751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cef9c64-ece5-4ff6-be24-fa4c22186b9d admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:55.392699 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fd04f7-f0ae-4d3a-8f8c-3f3f176b9dd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:55.393355 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fd04f7-f0ae-4d3a-8f8c-3f3f176b9dd5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:55.393638 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fd04f7-f0ae-4d3a-8f8c-3f3f176b9dd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:55.393916 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fd04f7-f0ae-4d3a-8f8c-3f3f176b9dd5 admin admin] PATH_INFO: `/v3/projects/149fb4bc8e1a47ffa2c194edd8318b48/users/e5f7708ca6fa412ba938ca22932d5bb2/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:55.410054 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fd04f7-f0ae-4d3a-8f8c-3f3f176b9dd5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=149fb4bc8e1a47ffa2c194edd8318b48, user_id=e5f7708ca6fa412ba938ca22932d5bb2, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:55.413372 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fd04f7-f0ae-4d3a-8f8c-3f3f176b9dd5 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:55.427258 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1238/2435] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:05:55 2022] PUT /identity/v3/projects/149fb4bc8e1a47ffa2c194edd8318b48/users/e5f7708ca6fa412ba938ca22932d5bb2/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:55.432074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277e4a27-fc83-44f6-8b2c-e49a577cca79 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:55.442409 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277e4a27-fc83-44f6-8b2c-e49a577cca79 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:55.499926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a53b8f-8edd-42b9-8c6d-576c1f5050e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:55.500598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a53b8f-8edd-42b9-8c6d-576c1f5050e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:55.500881 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a53b8f-8edd-42b9-8c6d-576c1f5050e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:55.501159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a53b8f-8edd-42b9-8c6d-576c1f5050e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:55.501804 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a53b8f-8edd-42b9-8c6d-576c1f5050e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:55.503494 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a53b8f-8edd-42b9-8c6d-576c1f5050e1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:55.513602 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1198/2436] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:05:55 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:05:55.517748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fd04f7-f0ae-4d3a-8f8c-3f3f176b9dd5 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:55.529117 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fd04f7-f0ae-4d3a-8f8c-3f3f176b9dd5 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:55.533706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f635fd17-5756-47f5-862e-efdf83633d50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:55.534388 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f635fd17-5756-47f5-862e-efdf83633d50 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:55.534662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f635fd17-5756-47f5-862e-efdf83633d50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:55.534961 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f635fd17-5756-47f5-862e-efdf83633d50 admin admin] PATH_INFO: `/v3/projects/149fb4bc8e1a47ffa2c194edd8318b48/users/e5f7708ca6fa412ba938ca22932d5bb2/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:55.537488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f635fd17-5756-47f5-862e-efdf83633d50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=149fb4bc8e1a47ffa2c194edd8318b48, user_id=e5f7708ca6fa412ba938ca22932d5bb2, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:55.540312 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f635fd17-5756-47f5-862e-efdf83633d50 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:55.551050 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1239/2437] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:05:55 2022] PUT /identity/v3/projects/149fb4bc8e1a47ffa2c194edd8318b48/users/e5f7708ca6fa412ba938ca22932d5bb2/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:05:55.561854 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a866b1-5436-4601-b815-fce0b38954d0 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:55.562118 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a866b1-5436-4601-b815-fce0b38954d0 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:55.562390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a866b1-5436-4601-b815-fce0b38954d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:55.591406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c6a866b1-5436-4601-b815-fce0b38954d0 None None] MFA Rules not processed for user `e5f7708ca6fa412ba938ca22932d5bb2`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:55.592591 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c6a866b1-5436-4601-b815-fce0b38954d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:55.661084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1199/2438] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:55 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:05:55.672385 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f635fd17-5756-47f5-862e-efdf83633d50 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:55.674105 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-f635fd17-5756-47f5-862e-efdf83633d50 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:55.732701 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f635fd17-5756-47f5-862e-efdf83633d50 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:55.736291 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3c9e84-3d7c-46f9-8b05-1783f723816e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:55.736897 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c9e84-3d7c-46f9-8b05-1783f723816e service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:55.737179 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c9e84-3d7c-46f9-8b05-1783f723816e service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:55.737445 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c9e84-3d7c-46f9-8b05-1783f723816e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:55.746698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3c9e84-3d7c-46f9-8b05-1783f723816e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:55.748161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3c9e84-3d7c-46f9-8b05-1783f723816e service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:55.758214 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1240/2439] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:05:55 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:05:56.355420 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a866b1-5436-4601-b815-fce0b38954d0 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:56.413932 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:56.435828 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a866b1-5436-4601-b815-fce0b38954d0 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:56.440590 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aac6ba6-0c47-4c4b-84ed-78d64ceeda1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:56.441395 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac6ba6-0c47-4c4b-84ed-78d64ceeda1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:56.441664 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac6ba6-0c47-4c4b-84ed-78d64ceeda1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:56.441945 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac6ba6-0c47-4c4b-84ed-78d64ceeda1a admin admin] PATH_INFO: `/v3/users/9db29765b5224a919e2143d82f710713` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:56.444199 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aac6ba6-0c47-4c4b-84ed-78d64ceeda1a admin admin] RBAC: Authorizing `identity:delete_user(user_id=9db29765b5224a919e2143d82f710713)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:56.446441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aac6ba6-0c47-4c4b-84ed-78d64ceeda1a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:56.501500 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4aac6ba6-0c47-4c4b-84ed-78d64ceeda1a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9db29765b5224a919e2143d82f710713'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:56.502431 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4aac6ba6-0c47-4c4b-84ed-78d64ceeda1a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9db29765b5224a919e2143d82f710713'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:56.512332 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4aac6ba6-0c47-4c4b-84ed-78d64ceeda1a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9db29765b5224a919e2143d82f710713'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:56.512985 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4aac6ba6-0c47-4c4b-84ed-78d64ceeda1a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9db29765b5224a919e2143d82f710713'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:56.525885 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-4aac6ba6-0c47-4c4b-84ed-78d64ceeda1a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9db29765b5224a919e2143d82f710713'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:56.538983 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1200/2440] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:05:56 2022] DELETE /identity/v3/users/9db29765b5224a919e2143d82f710713 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:56.543979 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9366e989-2945-4280-9baa-ce469a8d26de None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:56.543979 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9366e989-2945-4280-9baa-ce469a8d26de None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:56.544377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9366e989-2945-4280-9baa-ce469a8d26de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:56.564255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-9366e989-2945-4280-9baa-ce469a8d26de None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:56.565475 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-9366e989-2945-4280-9baa-ce469a8d26de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:56.631303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1241/2441] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:05:56.642514 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aac6ba6-0c47-4c4b-84ed-78d64ceeda1a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:56.644005 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4aac6ba6-0c47-4c4b-84ed-78d64ceeda1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:56.707430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aac6ba6-0c47-4c4b-84ed-78d64ceeda1a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:56.711584 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60dbf280-6bd9-454c-a12a-c029ffb05409 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:56.712212 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dbf280-6bd9-454c-a12a-c029ffb05409 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:56.712496 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dbf280-6bd9-454c-a12a-c029ffb05409 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:56.712765 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dbf280-6bd9-454c-a12a-c029ffb05409 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:56.722850 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60dbf280-6bd9-454c-a12a-c029ffb05409 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:56.724826 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60dbf280-6bd9-454c-a12a-c029ffb05409 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:56.737307 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1201/2442] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:05:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 01:05:57.035967 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9366e989-2945-4280-9baa-ce469a8d26de None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:57.038427 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-9366e989-2945-4280-9baa-ce469a8d26de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:57.060929 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9366e989-2945-4280-9baa-ce469a8d26de None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:57.066036 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ba1268-ee7a-43eb-85f1-0d21340739fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:57.066706 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ba1268-ee7a-43eb-85f1-0d21340739fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:57.067034 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ba1268-ee7a-43eb-85f1-0d21340739fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:57.067341 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ba1268-ee7a-43eb-85f1-0d21340739fa admin admin] PATH_INFO: `/v3/projects/5ea962541a1e43c9a6df32518a591d45` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:57.069069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ba1268-ee7a-43eb-85f1-0d21340739fa admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ea962541a1e43c9a6df32518a591d45)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:57.071334 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ba1268-ee7a-43eb-85f1-0d21340739fa admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:57.134157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-98ba1268-ee7a-43eb-85f1-0d21340739fa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ea962541a1e43c9a6df32518a591d45', 'request_id': 'req-98ba1268-ee7a-43eb-85f1-0d21340739fa'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:57.151045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:INSERT=5,keystone:SELECT=91,keystone:DELETE=4 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:57.151045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-98ba1268-ee7a-43eb-85f1-0d21340739fa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ea962541a1e43c9a6df32518a591d45', 'request_id': 'req-98ba1268-ee7a-43eb-85f1-0d21340739fa'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:57.156981 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-98ba1268-ee7a-43eb-85f1-0d21340739fa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ea962541a1e43c9a6df32518a591d45', 'request_id': 'req-98ba1268-ee7a-43eb-85f1-0d21340739fa'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:57.171375 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-98ba1268-ee7a-43eb-85f1-0d21340739fa admin admin] The token cache is being invalidate because project 5ea962541a1e43c9a6df32518a591d45 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:05:57.171811 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-98ba1268-ee7a-43eb-85f1-0d21340739fa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:57.172529 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-98ba1268-ee7a-43eb-85f1-0d21340739fa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:57.177197 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1242/2443] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:05:57 2022] DELETE /identity/v3/projects/5ea962541a1e43c9a6df32518a591d45 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:57.179259 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60dbf280-6bd9-454c-a12a-c029ffb05409 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:57.193300 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60dbf280-6bd9-454c-a12a-c029ffb05409 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:57.261945 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee0367c-1b46-4aeb-97bf-f9f35fb676cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:57.262694 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee0367c-1b46-4aeb-97bf-f9f35fb676cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:57.263249 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee0367c-1b46-4aeb-97bf-f9f35fb676cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:57.263612 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee0367c-1b46-4aeb-97bf-f9f35fb676cf admin admin] PATH_INFO: `/v3/users/e5f7708ca6fa412ba938ca22932d5bb2` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:57.264772 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee0367c-1b46-4aeb-97bf-f9f35fb676cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=e5f7708ca6fa412ba938ca22932d5bb2)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:57.266808 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee0367c-1b46-4aeb-97bf-f9f35fb676cf admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:57.273852 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:57.325798 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-aee0367c-1b46-4aeb-97bf-f9f35fb676cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e5f7708ca6fa412ba938ca22932d5bb2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:57.326513 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-aee0367c-1b46-4aeb-97bf-f9f35fb676cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e5f7708ca6fa412ba938ca22932d5bb2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:57.335362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-aee0367c-1b46-4aeb-97bf-f9f35fb676cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e5f7708ca6fa412ba938ca22932d5bb2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:57.336000 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-aee0367c-1b46-4aeb-97bf-f9f35fb676cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e5f7708ca6fa412ba938ca22932d5bb2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:57.348868 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-aee0367c-1b46-4aeb-97bf-f9f35fb676cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e5f7708ca6fa412ba938ca22932d5bb2'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:57.361373 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1202/2444] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:05:57 2022] DELETE /identity/v3/users/e5f7708ca6fa412ba938ca22932d5bb2 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:57.656035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ba1268-ee7a-43eb-85f1-0d21340739fa admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:57.657491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-98ba1268-ee7a-43eb-85f1-0d21340739fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:57.672481 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ba1268-ee7a-43eb-85f1-0d21340739fa admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:57.742075 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088936ba-8788-427a-978d-f316cb7db934 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:57.742615 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088936ba-8788-427a-978d-f316cb7db934 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:57.742883 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088936ba-8788-427a-978d-f316cb7db934 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:57.743183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088936ba-8788-427a-978d-f316cb7db934 admin admin] PATH_INFO: `/v3/projects/149fb4bc8e1a47ffa2c194edd8318b48` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:57.744386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088936ba-8788-427a-978d-f316cb7db934 admin admin] RBAC: Authorizing `identity:delete_project(project_id=149fb4bc8e1a47ffa2c194edd8318b48)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:57.746353 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088936ba-8788-427a-978d-f316cb7db934 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:57.799350 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-088936ba-8788-427a-978d-f316cb7db934 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '149fb4bc8e1a47ffa2c194edd8318b48', 'request_id': 'req-088936ba-8788-427a-978d-f316cb7db934'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:57.810534 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-088936ba-8788-427a-978d-f316cb7db934 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '149fb4bc8e1a47ffa2c194edd8318b48', 'request_id': 'req-088936ba-8788-427a-978d-f316cb7db934'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:57.818360 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-088936ba-8788-427a-978d-f316cb7db934 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '149fb4bc8e1a47ffa2c194edd8318b48', 'request_id': 'req-088936ba-8788-427a-978d-f316cb7db934'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:57.831032 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-088936ba-8788-427a-978d-f316cb7db934 admin admin] The token cache is being invalidate because project 149fb4bc8e1a47ffa2c194edd8318b48 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:05:57.831331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-088936ba-8788-427a-978d-f316cb7db934 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:57.832009 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-088936ba-8788-427a-978d-f316cb7db934 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:05:57.834127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1243/2445] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:05:57 2022] DELETE /identity/v3/projects/149fb4bc8e1a47ffa2c194edd8318b48 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:57.839617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197f3850-6ad1-4aff-a855-67c6cfd41539 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:57.839943 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197f3850-6ad1-4aff-a855-67c6cfd41539 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:57.840263 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197f3850-6ad1-4aff-a855-67c6cfd41539 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:57.863040 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-197f3850-6ad1-4aff-a855-67c6cfd41539 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:57.864293 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-197f3850-6ad1-4aff-a855-67c6cfd41539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:57.937589 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1203/2446] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:05:58.007199 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f773a323-f629-43b0-96b4-cd62e6e78705 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:58.007430 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f773a323-f629-43b0-96b4-cd62e6e78705 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:58.007670 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f773a323-f629-43b0-96b4-cd62e6e78705 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:58.029250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-f773a323-f629-43b0-96b4-cd62e6e78705 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:58.030500 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-f773a323-f629-43b0-96b4-cd62e6e78705 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:58.039853 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1244/2447] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:05:58.044147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197f3850-6ad1-4aff-a855-67c6cfd41539 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:58.054392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197f3850-6ad1-4aff-a855-67c6cfd41539 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:58.059306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99eb9ce8-f9f0-4cf0-9c49-1782bc24a08c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:58.059951 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99eb9ce8-f9f0-4cf0-9c49-1782bc24a08c admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:58.060227 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99eb9ce8-f9f0-4cf0-9c49-1782bc24a08c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:58.060495 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99eb9ce8-f9f0-4cf0-9c49-1782bc24a08c admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:58.061141 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99eb9ce8-f9f0-4cf0-9c49-1782bc24a08c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:05:58.061442 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99eb9ce8-f9f0-4cf0-9c49-1782bc24a08c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:58.063106 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99eb9ce8-f9f0-4cf0-9c49-1782bc24a08c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:58.076828 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1204/2448] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:05:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:05:58.081704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f773a323-f629-43b0-96b4-cd62e6e78705 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:58.094086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f773a323-f629-43b0-96b4-cd62e6e78705 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:58.101155 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7c356e-544c-49a5-a2eb-00f1b9aa41f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:58.102241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7c356e-544c-49a5-a2eb-00f1b9aa41f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:58.102750 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7c356e-544c-49a5-a2eb-00f1b9aa41f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:58.103318 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7c356e-544c-49a5-a2eb-00f1b9aa41f6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:58.104307 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7c356e-544c-49a5-a2eb-00f1b9aa41f6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:58.108218 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7c356e-544c-49a5-a2eb-00f1b9aa41f6 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:58.147495 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1245/2449] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:05:58 2022] POST /identity/v3/projects => generated 375 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:05:58.152288 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99eb9ce8-f9f0-4cf0-9c49-1782bc24a08c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:58.163233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99eb9ce8-f9f0-4cf0-9c49-1782bc24a08c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:58.229648 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d16fc1-e952-49f6-9be7-b9436a40e90f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:58.230289 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d16fc1-e952-49f6-9be7-b9436a40e90f admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:58.230567 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d16fc1-e952-49f6-9be7-b9436a40e90f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:58.230911 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d16fc1-e952-49f6-9be7-b9436a40e90f admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:58.231502 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d16fc1-e952-49f6-9be7-b9436a40e90f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:58.233522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d16fc1-e952-49f6-9be7-b9436a40e90f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:58.258716 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1205/2450] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:05:58 2022] POST /identity/v3/users => generated 288 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:05:58.264953 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7c356e-544c-49a5-a2eb-00f1b9aa41f6 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:58.277047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7c356e-544c-49a5-a2eb-00f1b9aa41f6 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:58.282200 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871c8d0c-15bd-40d3-bead-28847c0466b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:58.282939 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871c8d0c-15bd-40d3-bead-28847c0466b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:58.283315 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871c8d0c-15bd-40d3-bead-28847c0466b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:58.283661 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871c8d0c-15bd-40d3-bead-28847c0466b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:58.284392 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871c8d0c-15bd-40d3-bead-28847c0466b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:58.286084 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871c8d0c-15bd-40d3-bead-28847c0466b3 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:58.298900 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1246/2451] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:05:58 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:05:58.302570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d16fc1-e952-49f6-9be7-b9436a40e90f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:58.308149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:58.314584 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d16fc1-e952-49f6-9be7-b9436a40e90f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:58.324417 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce54765b-94cf-49e4-8ad9-bae677d43e68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:58.325398 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce54765b-94cf-49e4-8ad9-bae677d43e68 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:58.325868 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce54765b-94cf-49e4-8ad9-bae677d43e68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:58.326388 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce54765b-94cf-49e4-8ad9-bae677d43e68 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:58.327157 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce54765b-94cf-49e4-8ad9-bae677d43e68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:58.328905 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce54765b-94cf-49e4-8ad9-bae677d43e68 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:58.342286 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1206/2452] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:05:58 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:05:58.348785 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871c8d0c-15bd-40d3-bead-28847c0466b3 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:58.367178 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871c8d0c-15bd-40d3-bead-28847c0466b3 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:58.374062 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bf8cd5-1070-4191-9b14-2dd7ea9e0b12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:58.375271 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bf8cd5-1070-4191-9b14-2dd7ea9e0b12 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:58.375787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bf8cd5-1070-4191-9b14-2dd7ea9e0b12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:58.376288 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bf8cd5-1070-4191-9b14-2dd7ea9e0b12 admin admin] PATH_INFO: `/v3/projects/df4780ffda6e4ce39233f4d208f52c3e/users/e014023db7ea4ce4b83c818dbef0fbf0/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:58.402829 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95bf8cd5-1070-4191-9b14-2dd7ea9e0b12 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df4780ffda6e4ce39233f4d208f52c3e, user_id=e014023db7ea4ce4b83c818dbef0fbf0, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:58.405863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95bf8cd5-1070-4191-9b14-2dd7ea9e0b12 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:58.422021 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1247/2453] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:05:58 2022] PUT /identity/v3/projects/df4780ffda6e4ce39233f4d208f52c3e/users/e014023db7ea4ce4b83c818dbef0fbf0/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:05:58.433330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6248aa-93ae-4234-bef8-b11d90950f28 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:58.433789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6248aa-93ae-4234-bef8-b11d90950f28 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:58.433892 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6248aa-93ae-4234-bef8-b11d90950f28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:58.465835 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-7c6248aa-93ae-4234-bef8-b11d90950f28 None None] MFA Rules not processed for user `e014023db7ea4ce4b83c818dbef0fbf0`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:05:58.467103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7c6248aa-93ae-4234-bef8-b11d90950f28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:58.543686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1207/2454] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:05:58 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:05:58.557140 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bf8cd5-1070-4191-9b14-2dd7ea9e0b12 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:05:58.559239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-95bf8cd5-1070-4191-9b14-2dd7ea9e0b12 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:05:58.567525 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:INSERT=4,keystone:DELETE=10,keystone:SELECT=86 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:05:58.639897 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bf8cd5-1070-4191-9b14-2dd7ea9e0b12 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:05:58.644082 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f01eb84-4d96-4d8b-a3df-8ba37e2b9466 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:05:58.644727 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f01eb84-4d96-4d8b-a3df-8ba37e2b9466 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:05:58.645084 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f01eb84-4d96-4d8b-a3df-8ba37e2b9466 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:05:58.645366 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f01eb84-4d96-4d8b-a3df-8ba37e2b9466 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:05:58.659340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f01eb84-4d96-4d8b-a3df-8ba37e2b9466 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:05:58.661208 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f01eb84-4d96-4d8b-a3df-8ba37e2b9466 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:05:58.674853 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1248/2455] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:05:58 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:06:02.258578 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6248aa-93ae-4234-bef8-b11d90950f28 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:06:02.260137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-7c6248aa-93ae-4234-bef8-b11d90950f28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:06:02.326775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6248aa-93ae-4234-bef8-b11d90950f28 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:06:02.330621 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bdaed0-ec16-4140-844b-352b5f519be6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:06:02.331298 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bdaed0-ec16-4140-844b-352b5f519be6 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:02.331582 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bdaed0-ec16-4140-844b-352b5f519be6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:02.331867 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bdaed0-ec16-4140-844b-352b5f519be6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:02.334416 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-69bdaed0-ec16-4140-844b-352b5f519be6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:06:02.420375 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bdaed0-ec16-4140-844b-352b5f519be6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:06:02.423273 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bdaed0-ec16-4140-844b-352b5f519be6 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:06:02.442798 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1208/2456] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:06:02 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 01:06:08.670406 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=24 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:06:09.141298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f01eb84-4d96-4d8b-a3df-8ba37e2b9466 service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:06:09.202978 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f01eb84-4d96-4d8b-a3df-8ba37e2b9466 service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:06:09.207513 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1614148b-6467-4f3f-bea9-a96207b4cdb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:06:09.208138 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1614148b-6467-4f3f-bea9-a96207b4cdb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:09.208418 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1614148b-6467-4f3f-bea9-a96207b4cdb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:09.208690 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1614148b-6467-4f3f-bea9-a96207b4cdb2 admin admin] PATH_INFO: `/v3/users/e014023db7ea4ce4b83c818dbef0fbf0` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:09.209821 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1614148b-6467-4f3f-bea9-a96207b4cdb2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e014023db7ea4ce4b83c818dbef0fbf0)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:06:09.211792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1614148b-6467-4f3f-bea9-a96207b4cdb2 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:06:09.258267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1614148b-6467-4f3f-bea9-a96207b4cdb2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e014023db7ea4ce4b83c818dbef0fbf0'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:06:09.258961 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1614148b-6467-4f3f-bea9-a96207b4cdb2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e014023db7ea4ce4b83c818dbef0fbf0'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:06:09.266435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1614148b-6467-4f3f-bea9-a96207b4cdb2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e014023db7ea4ce4b83c818dbef0fbf0'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:06:09.278641 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1614148b-6467-4f3f-bea9-a96207b4cdb2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e014023db7ea4ce4b83c818dbef0fbf0'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:06:09.285524 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1614148b-6467-4f3f-bea9-a96207b4cdb2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e014023db7ea4ce4b83c818dbef0fbf0'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:06:09.291499 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1249/2457] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:06:09 2022] DELETE /identity/v3/users/e014023db7ea4ce4b83c818dbef0fbf0 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:06:09.296086 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112371d6-d281-4f00-9617-b099bdaaf6f9 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:09.296607 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112371d6-d281-4f00-9617-b099bdaaf6f9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:09.296729 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112371d6-d281-4f00-9617-b099bdaaf6f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:09.316903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-112371d6-d281-4f00-9617-b099bdaaf6f9 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:06:09.318625 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-112371d6-d281-4f00-9617-b099bdaaf6f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:06:09.344338 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:06:09.378142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1209/2458] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:06:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:06:09.387867 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1614148b-6467-4f3f-bea9-a96207b4cdb2 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:06:09.389285 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-1614148b-6467-4f3f-bea9-a96207b4cdb2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:06:09.448891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1614148b-6467-4f3f-bea9-a96207b4cdb2 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:06:09.452728 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0400e581-4bf6-44d4-80d0-9416cec6751b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:06:09.453429 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0400e581-4bf6-44d4-80d0-9416cec6751b service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:09.453764 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0400e581-4bf6-44d4-80d0-9416cec6751b service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:09.454044 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0400e581-4bf6-44d4-80d0-9416cec6751b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:09.463251 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0400e581-4bf6-44d4-80d0-9416cec6751b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:06:09.464691 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0400e581-4bf6-44d4-80d0-9416cec6751b service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:06:09.474698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1250/2459] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:06:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 01:06:09.748888 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112371d6-d281-4f00-9617-b099bdaaf6f9 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:06:09.750402 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-112371d6-d281-4f00-9617-b099bdaaf6f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:06:09.763896 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112371d6-d281-4f00-9617-b099bdaaf6f9 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:06:09.769080 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bebff1-d78d-47e0-8acf-64ae53e3a9f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:06:09.769738 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bebff1-d78d-47e0-8acf-64ae53e3a9f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:09.770060 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bebff1-d78d-47e0-8acf-64ae53e3a9f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:09.770341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bebff1-d78d-47e0-8acf-64ae53e3a9f1 admin admin] PATH_INFO: `/v3/projects/df4780ffda6e4ce39233f4d208f52c3e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:09.772350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bebff1-d78d-47e0-8acf-64ae53e3a9f1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=df4780ffda6e4ce39233f4d208f52c3e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:06:09.774546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bebff1-d78d-47e0-8acf-64ae53e3a9f1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:06:09.838750 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-68bebff1-d78d-47e0-8acf-64ae53e3a9f1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df4780ffda6e4ce39233f4d208f52c3e', 'request_id': 'req-68bebff1-d78d-47e0-8acf-64ae53e3a9f1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:06:09.847077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-68bebff1-d78d-47e0-8acf-64ae53e3a9f1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df4780ffda6e4ce39233f4d208f52c3e', 'request_id': 'req-68bebff1-d78d-47e0-8acf-64ae53e3a9f1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:06:09.854347 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-68bebff1-d78d-47e0-8acf-64ae53e3a9f1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df4780ffda6e4ce39233f4d208f52c3e', 'request_id': 'req-68bebff1-d78d-47e0-8acf-64ae53e3a9f1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:06:09.865186 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-68bebff1-d78d-47e0-8acf-64ae53e3a9f1 admin admin] The token cache is being invalidate because project df4780ffda6e4ce39233f4d208f52c3e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:06:09.865472 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-68bebff1-d78d-47e0-8acf-64ae53e3a9f1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:06:09.866138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-68bebff1-d78d-47e0-8acf-64ae53e3a9f1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:06:09.867649 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1210/2460] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:06:09 2022] DELETE /identity/v3/projects/df4780ffda6e4ce39233f4d208f52c3e => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:06:09.872537 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756a6ecf-82cd-4169-a580-c71c0cab5e3b None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:09.873009 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756a6ecf-82cd-4169-a580-c71c0cab5e3b None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:09.873109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756a6ecf-82cd-4169-a580-c71c0cab5e3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:09.892570 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-756a6ecf-82cd-4169-a580-c71c0cab5e3b None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:06:09.893784 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-756a6ecf-82cd-4169-a580-c71c0cab5e3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:06:09.912553 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:06:09.955836 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1251/2461] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:06:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:06:09.965325 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a68cc3-90ee-4f7e-b0c0-19b5462b3504 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:09.965615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a68cc3-90ee-4f7e-b0c0-19b5462b3504 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:09.965890 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a68cc3-90ee-4f7e-b0c0-19b5462b3504 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:09.988665 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b4a68cc3-90ee-4f7e-b0c0-19b5462b3504 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:06:09.990285 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b4a68cc3-90ee-4f7e-b0c0-19b5462b3504 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:06:09.998203 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1211/2462] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:06:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:06:10.002607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756a6ecf-82cd-4169-a580-c71c0cab5e3b None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:06:10.012838 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756a6ecf-82cd-4169-a580-c71c0cab5e3b None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:06:10.017452 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86904f5-02c1-48b5-9f53-47c0b81e37ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:06:10.018063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86904f5-02c1-48b5-9f53-47c0b81e37ee admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:10.018345 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86904f5-02c1-48b5-9f53-47c0b81e37ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:10.018617 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86904f5-02c1-48b5-9f53-47c0b81e37ee admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:10.019321 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86904f5-02c1-48b5-9f53-47c0b81e37ee admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:06:10.019626 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86904f5-02c1-48b5-9f53-47c0b81e37ee admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:06:10.022056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86904f5-02c1-48b5-9f53-47c0b81e37ee admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:06:10.034163 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1252/2463] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:06:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:06:10.039312 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a68cc3-90ee-4f7e-b0c0-19b5462b3504 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:06:10.049598 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a68cc3-90ee-4f7e-b0c0-19b5462b3504 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:06:10.057919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba1fa94-69d4-490e-8fae-3a59068e9319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:06:10.058756 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba1fa94-69d4-490e-8fae-3a59068e9319 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:10.059076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba1fa94-69d4-490e-8fae-3a59068e9319 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:10.059349 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba1fa94-69d4-490e-8fae-3a59068e9319 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:10.060091 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba1fa94-69d4-490e-8fae-3a59068e9319 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:06:10.062527 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba1fa94-69d4-490e-8fae-3a59068e9319 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:06:10.091184 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1212/2464] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:06:10 2022] POST /identity/v3/projects => generated 381 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:06:10.095872 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86904f5-02c1-48b5-9f53-47c0b81e37ee admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:06:10.106477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86904f5-02c1-48b5-9f53-47c0b81e37ee admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:06:10.168685 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11ef019-e581-4db9-94d4-5bece235c45a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:06:10.169295 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11ef019-e581-4db9-94d4-5bece235c45a admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:10.169582 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11ef019-e581-4db9-94d4-5bece235c45a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:10.169857 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11ef019-e581-4db9-94d4-5bece235c45a admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:10.170510 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11ef019-e581-4db9-94d4-5bece235c45a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:06:10.172463 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11ef019-e581-4db9-94d4-5bece235c45a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:06:10.194446 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1253/2465] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:06:10 2022] POST /identity/v3/users => generated 291 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:06:10.198894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba1fa94-69d4-490e-8fae-3a59068e9319 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:06:10.211664 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba1fa94-69d4-490e-8fae-3a59068e9319 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:06:10.219017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7a5c12-5dc2-4ccb-8b69-8dec6c0f0668 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:06:10.219629 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7a5c12-5dc2-4ccb-8b69-8dec6c0f0668 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:10.219909 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7a5c12-5dc2-4ccb-8b69-8dec6c0f0668 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:10.220182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7a5c12-5dc2-4ccb-8b69-8dec6c0f0668 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:10.220830 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7a5c12-5dc2-4ccb-8b69-8dec6c0f0668 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:06:10.222581 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7a5c12-5dc2-4ccb-8b69-8dec6c0f0668 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:06:10.233729 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1213/2466] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:06:10 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:06:10.237916 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11ef019-e581-4db9-94d4-5bece235c45a admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:06:10.248709 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11ef019-e581-4db9-94d4-5bece235c45a admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:06:10.253422 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf9a424-c87d-421c-ab15-ae2fb89b26c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:06:10.254020 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf9a424-c87d-421c-ab15-ae2fb89b26c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:10.254300 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf9a424-c87d-421c-ab15-ae2fb89b26c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:10.254575 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf9a424-c87d-421c-ab15-ae2fb89b26c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:10.256134 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf9a424-c87d-421c-ab15-ae2fb89b26c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:06:10.257744 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf9a424-c87d-421c-ab15-ae2fb89b26c7 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:06:10.268328 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1254/2467] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:06:10 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:06:10.273088 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7a5c12-5dc2-4ccb-8b69-8dec6c0f0668 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:06:10.283218 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7a5c12-5dc2-4ccb-8b69-8dec6c0f0668 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:06:10.288053 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92230225-eeec-4edb-b308-4d3d9ab266e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:06:10.288701 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92230225-eeec-4edb-b308-4d3d9ab266e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:10.289005 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92230225-eeec-4edb-b308-4d3d9ab266e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:10.289307 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92230225-eeec-4edb-b308-4d3d9ab266e8 admin admin] PATH_INFO: `/v3/projects/c322a47bb2364c8ea9250a7ecea537bf/users/602ff9dfc79d43839abbb179cbf6f9fd/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:10.304142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92230225-eeec-4edb-b308-4d3d9ab266e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c322a47bb2364c8ea9250a7ecea537bf, user_id=602ff9dfc79d43839abbb179cbf6f9fd, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:06:10.306842 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92230225-eeec-4edb-b308-4d3d9ab266e8 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:06:10.321141 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1214/2468] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:06:10 2022] PUT /identity/v3/projects/c322a47bb2364c8ea9250a7ecea537bf/users/602ff9dfc79d43839abbb179cbf6f9fd/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:06:10.326184 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c64e90-e585-4d27-b358-cd2d1c7f7fa9 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:10.326477 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c64e90-e585-4d27-b358-cd2d1c7f7fa9 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:10.326756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c64e90-e585-4d27-b358-cd2d1c7f7fa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:10.346802 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-a2c64e90-e585-4d27-b358-cd2d1c7f7fa9 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:06:10.348046 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-a2c64e90-e585-4d27-b358-cd2d1c7f7fa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:06:10.407846 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1255/2469] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:06:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:06:10.418137 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92230225-eeec-4edb-b308-4d3d9ab266e8 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:06:10.419858 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-92230225-eeec-4edb-b308-4d3d9ab266e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:06:10.476291 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92230225-eeec-4edb-b308-4d3d9ab266e8 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:06:10.479853 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa22a90-18c1-405e-ae7e-5effe597ec01 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:06:10.480459 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa22a90-18c1-405e-ae7e-5effe597ec01 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:10.480730 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa22a90-18c1-405e-ae7e-5effe597ec01 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:10.480998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa22a90-18c1-405e-ae7e-5effe597ec01 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:10.486760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:06:10.491976 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa22a90-18c1-405e-ae7e-5effe597ec01 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:06:10.493926 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa22a90-18c1-405e-ae7e-5effe597ec01 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:06:10.504747 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1215/2470] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:06:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:06:11.135826 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c39999-330d-4615-9ef4-9d1beb9d7b12 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:11.136386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c39999-330d-4615-9ef4-9d1beb9d7b12 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:11.136491 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c39999-330d-4615-9ef4-9d1beb9d7b12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:11.170038 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-97c39999-330d-4615-9ef4-9d1beb9d7b12 None None] MFA Rules not processed for user `602ff9dfc79d43839abbb179cbf6f9fd`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:06:11.171624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-97c39999-330d-4615-9ef4-9d1beb9d7b12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:06:11.190787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:06:11.246318 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1256/2471] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:06:11 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:06:11.257063 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa22a90-18c1-405e-ae7e-5effe597ec01 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:06:11.258392 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-4aa22a90-18c1-405e-ae7e-5effe597ec01 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:06:11.319607 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa22a90-18c1-405e-ae7e-5effe597ec01 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:06:11.323784 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec4dca6-4a08-4b6c-ac3a-1f5872914514 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:06:11.324395 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec4dca6-4a08-4b6c-ac3a-1f5872914514 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:11.324679 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec4dca6-4a08-4b6c-ac3a-1f5872914514 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:11.324955 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec4dca6-4a08-4b6c-ac3a-1f5872914514 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:11.334209 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec4dca6-4a08-4b6c-ac3a-1f5872914514 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:06:11.335711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec4dca6-4a08-4b6c-ac3a-1f5872914514 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:06:11.344566 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1216/2472] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:06:11 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:06:12.711441 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2af5abf-b60a-4d01-8353-10fc2569bafe None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:12.711985 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2af5abf-b60a-4d01-8353-10fc2569bafe None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:12.712101 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2af5abf-b60a-4d01-8353-10fc2569bafe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:12.732645 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-a2af5abf-b60a-4d01-8353-10fc2569bafe None None] MFA Rules not processed for user `602ff9dfc79d43839abbb179cbf6f9fd`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:06:12.734247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-a2af5abf-b60a-4d01-8353-10fc2569bafe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:06:12.741777 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1257/2473] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:06:12 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:06:21.340506 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=29 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:06:22.726301 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=22 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:06:28.267936 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da313261-0d39-442c-a347-37cd03d26c38 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:06:28.268483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da313261-0d39-442c-a347-37cd03d26c38 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:06:28.268483 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da313261-0d39-442c-a347-37cd03d26c38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:06:28.288522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-da313261-0d39-442c-a347-37cd03d26c38 None None] MFA Rules not processed for user `e1283100ac2c439cb308b9c57ae31f40`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:06:28.289919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-da313261-0d39-442c-a347-37cd03d26c38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:06:28.347638 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1217/2474] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:06:28 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:06:38.339622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=25 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:07:17.937036 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015edc5e-28bb-43c8-bfb7-fe6ad9a65dcb None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:07:17.937732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015edc5e-28bb-43c8-bfb7-fe6ad9a65dcb None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:07:17.937732 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015edc5e-28bb-43c8-bfb7-fe6ad9a65dcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:07:17.957901 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-015edc5e-28bb-43c8-bfb7-fe6ad9a65dcb None None] MFA Rules not processed for user `602ff9dfc79d43839abbb179cbf6f9fd`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:07:17.959127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-015edc5e-28bb-43c8-bfb7-fe6ad9a65dcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:07:17.966231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1258/2475] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:07:17 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:07:18.006924 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da313261-0d39-442c-a347-37cd03d26c38 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:07:18.008369 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-da313261-0d39-442c-a347-37cd03d26c38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:07:18.068299 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da313261-0d39-442c-a347-37cd03d26c38 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:07:18.072781 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f418df68-65be-4cd7-bf53-15c1024dd577 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '78473c29cef449b89bc9185de67f1bf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:07:18.073390 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f418df68-65be-4cd7-bf53-15c1024dd577 service glance] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:07:18.073671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f418df68-65be-4cd7-bf53-15c1024dd577 service glance] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:07:18.073947 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f418df68-65be-4cd7-bf53-15c1024dd577 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:07:18.075721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-f418df68-65be-4cd7-bf53-15c1024dd577 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:07:18.147949 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f418df68-65be-4cd7-bf53-15c1024dd577 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:07:18.149393 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f418df68-65be-4cd7-bf53-15c1024dd577 service glance] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:07:18.160405 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1218/2476] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:07:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:07:18.692120 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015edc5e-28bb-43c8-bfb7-fe6ad9a65dcb None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:07:18.704018 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015edc5e-28bb-43c8-bfb7-fe6ad9a65dcb None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:07:18.707756 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18341151-ac9b-45cd-b2ec-79e927827ac0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:07:18.708366 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18341151-ac9b-45cd-b2ec-79e927827ac0 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:07:18.708650 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18341151-ac9b-45cd-b2ec-79e927827ac0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:07:18.708925 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18341151-ac9b-45cd-b2ec-79e927827ac0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:07:18.710666 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-18341151-ac9b-45cd-b2ec-79e927827ac0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:07:18.726383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18341151-ac9b-45cd-b2ec-79e927827ac0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:07:18.728026 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18341151-ac9b-45cd-b2ec-79e927827ac0 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:07:18.742483 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1259/2477] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:07:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 16 01:07:28.155524 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=54 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:07:28.736972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=9 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:07:56.371447 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f418df68-65be-4cd7-bf53-15c1024dd577 service glance] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:07:56.381821 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f418df68-65be-4cd7-bf53-15c1024dd577 service glance] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:07:56.385902 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bc65d7-42ce-4169-850d-da29ef4f3009 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:07:56.386539 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bc65d7-42ce-4169-850d-da29ef4f3009 service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:07:56.386804 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bc65d7-42ce-4169-850d-da29ef4f3009 service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:07:56.387120 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bc65d7-42ce-4169-850d-da29ef4f3009 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:07:56.388907 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b8bc65d7-42ce-4169-850d-da29ef4f3009 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:07:56.402122 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8bc65d7-42ce-4169-850d-da29ef4f3009 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:07:56.403614 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8bc65d7-42ce-4169-850d-da29ef4f3009 service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:07:56.414238 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1219/2478] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:07:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:08:03.816806 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18341151-ac9b-45cd-b2ec-79e927827ac0 service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:03.818697 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-18341151-ac9b-45cd-b2ec-79e927827ac0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:03.832880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18341151-ac9b-45cd-b2ec-79e927827ac0 service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:03.837624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eacaed2-bb96-444e-ad8a-99b503527d08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:03.838355 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eacaed2-bb96-444e-ad8a-99b503527d08 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:03.838847 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eacaed2-bb96-444e-ad8a-99b503527d08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:03.839161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eacaed2-bb96-444e-ad8a-99b503527d08 admin admin] PATH_INFO: `/v3/users/c4e65cef89af48a7a51e5548634de75f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:03.840303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eacaed2-bb96-444e-ad8a-99b503527d08 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c4e65cef89af48a7a51e5548634de75f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:03.842247 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eacaed2-bb96-444e-ad8a-99b503527d08 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:03.895378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1eacaed2-bb96-444e-ad8a-99b503527d08 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c4e65cef89af48a7a51e5548634de75f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:03.896041 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1eacaed2-bb96-444e-ad8a-99b503527d08 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c4e65cef89af48a7a51e5548634de75f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:03.905354 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1eacaed2-bb96-444e-ad8a-99b503527d08 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c4e65cef89af48a7a51e5548634de75f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:03.920752 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1eacaed2-bb96-444e-ad8a-99b503527d08 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c4e65cef89af48a7a51e5548634de75f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:03.927799 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1eacaed2-bb96-444e-ad8a-99b503527d08 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c4e65cef89af48a7a51e5548634de75f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:03.934308 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1260/2479] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:08:03 2022] DELETE /identity/v3/users/c4e65cef89af48a7a51e5548634de75f => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:04.117863 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bc65d7-42ce-4169-850d-da29ef4f3009 service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:04.135072 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bc65d7-42ce-4169-850d-da29ef4f3009 service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:04.206428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:04.206428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:04.206762 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:04.206762 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758 admin admin] PATH_INFO: `/v3/projects/cb1743385ea14cc39244c83e8fc88e7e` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:04.207890 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb1743385ea14cc39244c83e8fc88e7e)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:04.209815 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:04.260998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb1743385ea14cc39244c83e8fc88e7e', 'request_id': 'req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:04.267607 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb1743385ea14cc39244c83e8fc88e7e', 'request_id': 'req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:04.274292 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb1743385ea14cc39244c83e8fc88e7e', 'request_id': 'req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:04.285461 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758 admin admin] The token cache is being invalidate because project cb1743385ea14cc39244c83e8fc88e7e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:08:04.285771 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:04.286428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:04.287982 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1220/2480] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:08:04 2022] DELETE /identity/v3/projects/cb1743385ea14cc39244c83e8fc88e7e => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:04.292616 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eacaed2-bb96-444e-ad8a-99b503527d08 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:04.294064 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-1eacaed2-bb96-444e-ad8a-99b503527d08 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:04.307607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eacaed2-bb96-444e-ad8a-99b503527d08 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:04.368120 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5af731-85bf-4e85-958c-7c25dd0eb214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:04.368807 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5af731-85bf-4e85-958c-7c25dd0eb214 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:04.369056 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5af731-85bf-4e85-958c-7c25dd0eb214 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:04.369445 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5af731-85bf-4e85-958c-7c25dd0eb214 admin admin] PATH_INFO: `/v3/users/e4c1f5685bf840588d90104aa11e4920` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:04.370468 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5af731-85bf-4e85-958c-7c25dd0eb214 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4c1f5685bf840588d90104aa11e4920)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:04.372440 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5af731-85bf-4e85-958c-7c25dd0eb214 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:04.430796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3a5af731-85bf-4e85-958c-7c25dd0eb214 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4c1f5685bf840588d90104aa11e4920'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:04.432084 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3a5af731-85bf-4e85-958c-7c25dd0eb214 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4c1f5685bf840588d90104aa11e4920'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:04.440076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3a5af731-85bf-4e85-958c-7c25dd0eb214 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4c1f5685bf840588d90104aa11e4920'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:04.456400 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3a5af731-85bf-4e85-958c-7c25dd0eb214 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4c1f5685bf840588d90104aa11e4920'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:04.459379 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:04.467052 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-3a5af731-85bf-4e85-958c-7c25dd0eb214 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4c1f5685bf840588d90104aa11e4920'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:04.479173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1261/2481] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:08:04 2022] DELETE /identity/v3/users/e4c1f5685bf840588d90104aa11e4920 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:04.687014 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:04.688877 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:04.709264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3003d5d6-4abb-4c22-bb17-bdf8c6c41758 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:04.778743 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24b5bac-e20b-471c-9f2e-838b78d077c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:04.779444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24b5bac-e20b-471c-9f2e-838b78d077c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:04.779735 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24b5bac-e20b-471c-9f2e-838b78d077c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:04.780111 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24b5bac-e20b-471c-9f2e-838b78d077c5 admin admin] PATH_INFO: `/v3/projects/403b173e9ff04603b1c33837dcd1ba73` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:04.781349 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24b5bac-e20b-471c-9f2e-838b78d077c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=403b173e9ff04603b1c33837dcd1ba73)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:04.783471 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24b5bac-e20b-471c-9f2e-838b78d077c5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:04.808196 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:04.836933 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e24b5bac-e20b-471c-9f2e-838b78d077c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '403b173e9ff04603b1c33837dcd1ba73', 'request_id': 'req-e24b5bac-e20b-471c-9f2e-838b78d077c5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:04.845050 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e24b5bac-e20b-471c-9f2e-838b78d077c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '403b173e9ff04603b1c33837dcd1ba73', 'request_id': 'req-e24b5bac-e20b-471c-9f2e-838b78d077c5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:04.852198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e24b5bac-e20b-471c-9f2e-838b78d077c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '403b173e9ff04603b1c33837dcd1ba73', 'request_id': 'req-e24b5bac-e20b-471c-9f2e-838b78d077c5'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:04.864714 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e24b5bac-e20b-471c-9f2e-838b78d077c5 admin admin] The token cache is being invalidate because project 403b173e9ff04603b1c33837dcd1ba73 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:08:04.865021 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e24b5bac-e20b-471c-9f2e-838b78d077c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:04.865802 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e24b5bac-e20b-471c-9f2e-838b78d077c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:04.867358 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1221/2482] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:08:04 2022] DELETE /identity/v3/projects/403b173e9ff04603b1c33837dcd1ba73 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:07.379902 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5af731-85bf-4e85-958c-7c25dd0eb214 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:07.381430 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-3a5af731-85bf-4e85-958c-7c25dd0eb214 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:07.456736 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5af731-85bf-4e85-958c-7c25dd0eb214 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:07.518747 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f67ed6-1a8a-455e-b886-533fd40f0363 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:07.519379 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f67ed6-1a8a-455e-b886-533fd40f0363 service nova] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:07.519657 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f67ed6-1a8a-455e-b886-533fd40f0363 service nova] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:07.519927 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f67ed6-1a8a-455e-b886-533fd40f0363 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:07.521649 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-f9f67ed6-1a8a-455e-b886-533fd40f0363 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:07.580570 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f67ed6-1a8a-455e-b886-533fd40f0363 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:07.582181 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f67ed6-1a8a-455e-b886-533fd40f0363 service nova] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:07.591976 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1262/2483] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:08:07 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:08:10.598454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24b5bac-e20b-471c-9f2e-838b78d077c5 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:10.600639 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e24b5bac-e20b-471c-9f2e-838b78d077c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:10.617703 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24b5bac-e20b-471c-9f2e-838b78d077c5 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:10.624692 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315d57b1-6965-4574-a313-b80ceb6061a0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eba26813e154595a12a67fe4be5eeff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:10.625335 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315d57b1-6965-4574-a313-b80ceb6061a0 service nova] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:10.625611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315d57b1-6965-4574-a313-b80ceb6061a0 service nova] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:10.625879 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315d57b1-6965-4574-a313-b80ceb6061a0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:10.627863 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-315d57b1-6965-4574-a313-b80ceb6061a0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:10.646874 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315d57b1-6965-4574-a313-b80ceb6061a0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:10.648690 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315d57b1-6965-4574-a313-b80ceb6061a0 service nova] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:10.660629 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1222/2484] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:08:10 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:08:16.438426 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f67ed6-1a8a-455e-b886-533fd40f0363 service nova] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:16.439832 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-f9f67ed6-1a8a-455e-b886-533fd40f0363 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:16.471488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=100 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:16.500659 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f67ed6-1a8a-455e-b886-533fd40f0363 service nova] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:16.504609 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdacec6-3502-4ef1-b499-4e0503993547 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:16.505222 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdacec6-3502-4ef1-b499-4e0503993547 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:16.505535 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdacec6-3502-4ef1-b499-4e0503993547 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:16.505811 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdacec6-3502-4ef1-b499-4e0503993547 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:16.507567 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-1bdacec6-3502-4ef1-b499-4e0503993547 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:16.573476 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdacec6-3502-4ef1-b499-4e0503993547 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:16.574944 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdacec6-3502-4ef1-b499-4e0503993547 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:16.584868 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1263/2485] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:08:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:08:18.859570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315d57b1-6965-4574-a313-b80ceb6061a0 service nova] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:18.861076 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-315d57b1-6965-4574-a313-b80ceb6061a0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:18.877138 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315d57b1-6965-4574-a313-b80ceb6061a0 service nova] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:18.881873 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87afe6d0-dbe2-48c8-bed8-bcd4db83393f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:18.882508 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87afe6d0-dbe2-48c8-bed8-bcd4db83393f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:18.882807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87afe6d0-dbe2-48c8-bed8-bcd4db83393f admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:18.883090 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87afe6d0-dbe2-48c8-bed8-bcd4db83393f admin admin] PATH_INFO: `/v3/users/602ff9dfc79d43839abbb179cbf6f9fd` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:18.884265 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87afe6d0-dbe2-48c8-bed8-bcd4db83393f admin admin] RBAC: Authorizing `identity:delete_user(user_id=602ff9dfc79d43839abbb179cbf6f9fd)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:18.886201 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87afe6d0-dbe2-48c8-bed8-bcd4db83393f admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:18.936582 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-87afe6d0-dbe2-48c8-bed8-bcd4db83393f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '602ff9dfc79d43839abbb179cbf6f9fd'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:18.937255 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-87afe6d0-dbe2-48c8-bed8-bcd4db83393f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '602ff9dfc79d43839abbb179cbf6f9fd'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:18.944782 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-87afe6d0-dbe2-48c8-bed8-bcd4db83393f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '602ff9dfc79d43839abbb179cbf6f9fd'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:18.945512 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-87afe6d0-dbe2-48c8-bed8-bcd4db83393f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '602ff9dfc79d43839abbb179cbf6f9fd'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:18.958325 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-87afe6d0-dbe2-48c8-bed8-bcd4db83393f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '602ff9dfc79d43839abbb179cbf6f9fd'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:18.972912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1223/2486] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:08:18 2022] DELETE /identity/v3/users/602ff9dfc79d43839abbb179cbf6f9fd => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:19.204156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdacec6-3502-4ef1-b499-4e0503993547 service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:19.215198 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdacec6-3502-4ef1-b499-4e0503993547 service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:19.278143 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:19.278784 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:19.279095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:19.279367 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc admin admin] PATH_INFO: `/v3/projects/c322a47bb2364c8ea9250a7ecea537bf` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:19.280513 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=c322a47bb2364c8ea9250a7ecea537bf)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:19.282431 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:19.331397 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c322a47bb2364c8ea9250a7ecea537bf', 'request_id': 'req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:19.339400 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c322a47bb2364c8ea9250a7ecea537bf', 'request_id': 'req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:19.345233 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c322a47bb2364c8ea9250a7ecea537bf', 'request_id': 'req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:19.357991 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc admin admin] The token cache is being invalidate because project c322a47bb2364c8ea9250a7ecea537bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:08:19.358471 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:19.359157 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:19.360975 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1264/2487] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:08:19 2022] DELETE /identity/v3/projects/c322a47bb2364c8ea9250a7ecea537bf => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:19.652551 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87afe6d0-dbe2-48c8-bed8-bcd4db83393f admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:19.654188 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-87afe6d0-dbe2-48c8-bed8-bcd4db83393f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:19.674149 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:19.675712 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:19.686481 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:19.753619 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa95fdd-9010-4a0e-b7a6-37cc04ff27dc admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:19.756182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87afe6d0-dbe2-48c8-bed8-bcd4db83393f admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:19.758817 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8771242-6700-405c-be3c-ca4c7d43b829 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:19.759492 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8771242-6700-405c-be3c-ca4c7d43b829 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:19.759867 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8771242-6700-405c-be3c-ca4c7d43b829 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:19.760076 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8771242-6700-405c-be3c-ca4c7d43b829 admin admin] PATH_INFO: `/v3/users/e1283100ac2c439cb308b9c57ae31f40` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:19.761197 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8771242-6700-405c-be3c-ca4c7d43b829 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1283100ac2c439cb308b9c57ae31f40)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:19.762326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:19.763298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8771242-6700-405c-be3c-ca4c7d43b829 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:19.816932 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c8771242-6700-405c-be3c-ca4c7d43b829 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1283100ac2c439cb308b9c57ae31f40'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:19.817838 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c8771242-6700-405c-be3c-ca4c7d43b829 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1283100ac2c439cb308b9c57ae31f40'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:19.830767 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c8771242-6700-405c-be3c-ca4c7d43b829 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1283100ac2c439cb308b9c57ae31f40'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:19.835822 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121bc861-cc11-4ba7-bd61-17223cb3f983 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87785ccd37a642e08a0ad34a4fb1c754', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:19.836995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121bc861-cc11-4ba7-bd61-17223cb3f983 service placement] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:19.837281 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121bc861-cc11-4ba7-bd61-17223cb3f983 service placement] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:19.837555 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121bc861-cc11-4ba7-bd61-17223cb3f983 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:19.839693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-121bc861-cc11-4ba7-bd61-17223cb3f983 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:19.846153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c8771242-6700-405c-be3c-ca4c7d43b829 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1283100ac2c439cb308b9c57ae31f40'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:19.854509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121bc861-cc11-4ba7-bd61-17223cb3f983 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:19.854950 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c8771242-6700-405c-be3c-ca4c7d43b829 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1283100ac2c439cb308b9c57ae31f40'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:19.856148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121bc861-cc11-4ba7-bd61-17223cb3f983 service placement] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:19.861951 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1265/2488] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:08:19 2022] DELETE /identity/v3/users/e1283100ac2c439cb308b9c57ae31f40 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:08:19.868873 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1224/2489] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:08:19 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:08:20.051096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8771242-6700-405c-be3c-ca4c7d43b829 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:20.052563 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c8771242-6700-405c-be3c-ca4c7d43b829 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:20.124332 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8771242-6700-405c-be3c-ca4c7d43b829 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:20.128899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29ebca0-ba4e-4641-a3af-04925186d319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:20.129528 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29ebca0-ba4e-4641-a3af-04925186d319 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:20.129809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29ebca0-ba4e-4641-a3af-04925186d319 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:20.130081 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29ebca0-ba4e-4641-a3af-04925186d319 admin admin] PATH_INFO: `/v3/projects/b8852bfe31aa4ba68190661d26c177cb` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:20.131293 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29ebca0-ba4e-4641-a3af-04925186d319 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8852bfe31aa4ba68190661d26c177cb)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:20.133221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29ebca0-ba4e-4641-a3af-04925186d319 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:20.165564 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:20.183072 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-f29ebca0-ba4e-4641-a3af-04925186d319 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8852bfe31aa4ba68190661d26c177cb', 'request_id': 'req-f29ebca0-ba4e-4641-a3af-04925186d319'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:20.190745 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-f29ebca0-ba4e-4641-a3af-04925186d319 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8852bfe31aa4ba68190661d26c177cb', 'request_id': 'req-f29ebca0-ba4e-4641-a3af-04925186d319'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:20.197809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-f29ebca0-ba4e-4641-a3af-04925186d319 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8852bfe31aa4ba68190661d26c177cb', 'request_id': 'req-f29ebca0-ba4e-4641-a3af-04925186d319'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:20.209576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-f29ebca0-ba4e-4641-a3af-04925186d319 admin admin] The token cache is being invalidate because project b8852bfe31aa4ba68190661d26c177cb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:08:20.210375 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-f29ebca0-ba4e-4641-a3af-04925186d319 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:20.210592 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-f29ebca0-ba4e-4641-a3af-04925186d319 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:20.212137 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1266/2490] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:08:20 2022] DELETE /identity/v3/projects/b8852bfe31aa4ba68190661d26c177cb => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:20.217134 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867ef61f-21a4-4f04-8eab-fed9d5e93544 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:20.217420 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867ef61f-21a4-4f04-8eab-fed9d5e93544 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:20.217695 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867ef61f-21a4-4f04-8eab-fed9d5e93544 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:20.238204 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-867ef61f-21a4-4f04-8eab-fed9d5e93544 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:08:20.239454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-867ef61f-21a4-4f04-8eab-fed9d5e93544 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:20.298967 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1225/2491] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:08:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:08:20.307453 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c600f1-7067-4658-b303-48a680275385 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:20.307773 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c600f1-7067-4658-b303-48a680275385 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:20.308067 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c600f1-7067-4658-b303-48a680275385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:20.328517 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-44c600f1-7067-4658-b303-48a680275385 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:08:20.329740 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-44c600f1-7067-4658-b303-48a680275385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:20.338009 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1267/2492] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:08:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:08:20.342232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867ef61f-21a4-4f04-8eab-fed9d5e93544 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:20.352499 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867ef61f-21a4-4f04-8eab-fed9d5e93544 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:20.357098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628f578f-2e06-41fb-bcbf-c7485b8828a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:20.357699 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628f578f-2e06-41fb-bcbf-c7485b8828a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:20.357987 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628f578f-2e06-41fb-bcbf-c7485b8828a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:20.358260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628f578f-2e06-41fb-bcbf-c7485b8828a2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:20.358930 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628f578f-2e06-41fb-bcbf-c7485b8828a2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:08:20.359242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628f578f-2e06-41fb-bcbf-c7485b8828a2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:20.360836 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628f578f-2e06-41fb-bcbf-c7485b8828a2 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:20.372859 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1226/2493] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:08:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:08:20.377158 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c600f1-7067-4658-b303-48a680275385 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:20.386576 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c600f1-7067-4658-b303-48a680275385 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:20.391167 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0703c52-fca5-4ca7-9bb1-924f2d0122c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:20.391760 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0703c52-fca5-4ca7-9bb1-924f2d0122c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:20.392059 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0703c52-fca5-4ca7-9bb1-924f2d0122c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:20.392330 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0703c52-fca5-4ca7-9bb1-924f2d0122c8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:20.392960 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0703c52-fca5-4ca7-9bb1-924f2d0122c8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:20.394898 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0703c52-fca5-4ca7-9bb1-924f2d0122c8 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:20.434912 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1268/2494] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:08:20 2022] POST /identity/v3/projects => generated 367 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:08:20.437129 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628f578f-2e06-41fb-bcbf-c7485b8828a2 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:20.447454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628f578f-2e06-41fb-bcbf-c7485b8828a2 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:20.503807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3d1432-fd5d-4e36-bbf1-880929b11310 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:20.504417 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3d1432-fd5d-4e36-bbf1-880929b11310 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:20.504696 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3d1432-fd5d-4e36-bbf1-880929b11310 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:20.504968 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3d1432-fd5d-4e36-bbf1-880929b11310 admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:20.505602 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3d1432-fd5d-4e36-bbf1-880929b11310 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:20.507633 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3d1432-fd5d-4e36-bbf1-880929b11310 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:20.528789 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1227/2495] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:08:20 2022] POST /identity/v3/users => generated 284 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:08:20.533008 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0703c52-fca5-4ca7-9bb1-924f2d0122c8 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:20.544067 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0703c52-fca5-4ca7-9bb1-924f2d0122c8 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:20.548703 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171b60f1-2053-4711-b466-189b80454bed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:20.549357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171b60f1-2053-4711-b466-189b80454bed admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:20.549587 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171b60f1-2053-4711-b466-189b80454bed admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:20.549866 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171b60f1-2053-4711-b466-189b80454bed admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:20.550536 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171b60f1-2053-4711-b466-189b80454bed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:20.552198 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171b60f1-2053-4711-b466-189b80454bed admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:20.562186 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1269/2496] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:08:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 01:08:20.566216 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3d1432-fd5d-4e36-bbf1-880929b11310 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:20.576812 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3d1432-fd5d-4e36-bbf1-880929b11310 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:20.581488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ffa8ca-559d-4e44-bea7-be4438000096 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:20.582109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ffa8ca-559d-4e44-bea7-be4438000096 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:20.582391 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ffa8ca-559d-4e44-bea7-be4438000096 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:20.582669 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ffa8ca-559d-4e44-bea7-be4438000096 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:20.583440 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ffa8ca-559d-4e44-bea7-be4438000096 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:20.585039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ffa8ca-559d-4e44-bea7-be4438000096 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:20.596317 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1228/2497] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:08:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:08:20.600465 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171b60f1-2053-4711-b466-189b80454bed admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:20.611063 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171b60f1-2053-4711-b466-189b80454bed admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:20.615619 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221434cc-d166-4eb9-adb0-728cbd606aa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:20.616259 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221434cc-d166-4eb9-adb0-728cbd606aa4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:20.616537 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221434cc-d166-4eb9-adb0-728cbd606aa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:20.616810 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221434cc-d166-4eb9-adb0-728cbd606aa4 admin admin] PATH_INFO: `/v3/projects/34db4209796f4261aa01c1f0175e0da0/users/4780eca202394e83aec122f9a5143482/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:20.631377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221434cc-d166-4eb9-adb0-728cbd606aa4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=34db4209796f4261aa01c1f0175e0da0, user_id=4780eca202394e83aec122f9a5143482, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:20.634231 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221434cc-d166-4eb9-adb0-728cbd606aa4 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:20.647239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1270/2498] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:08:20 2022] PUT /identity/v3/projects/34db4209796f4261aa01c1f0175e0da0/users/4780eca202394e83aec122f9a5143482/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:20.658228 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafdd041-cc33-4470-bb1e-cde9ac5bb006 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:20.658509 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafdd041-cc33-4470-bb1e-cde9ac5bb006 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:20.658780 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafdd041-cc33-4470-bb1e-cde9ac5bb006 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:20.667877 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:20.689979 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-cafdd041-cc33-4470-bb1e-cde9ac5bb006 None None] MFA Rules not processed for user `4780eca202394e83aec122f9a5143482`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:08:20.691223 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cafdd041-cc33-4470-bb1e-cde9ac5bb006 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:20.757244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1229/2499] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:08:20 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:08:20.768773 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221434cc-d166-4eb9-adb0-728cbd606aa4 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:20.770519 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-221434cc-d166-4eb9-adb0-728cbd606aa4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:20.828908 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221434cc-d166-4eb9-adb0-728cbd606aa4 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:20.832500 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae3fadd-52b3-4954-82e0-b89d30c756a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:20.833105 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae3fadd-52b3-4954-82e0-b89d30c756a2 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:20.833386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae3fadd-52b3-4954-82e0-b89d30c756a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:20.833662 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae3fadd-52b3-4954-82e0-b89d30c756a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:20.842998 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae3fadd-52b3-4954-82e0-b89d30c756a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:20.844444 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae3fadd-52b3-4954-82e0-b89d30c756a2 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:20.853532 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1271/2500] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:08:20 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:08:29.250188 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafdd041-cc33-4470-bb1e-cde9ac5bb006 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:29.312396 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafdd041-cc33-4470-bb1e-cde9ac5bb006 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:29.316904 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9eab228-a2fb-4718-9f49-b3b3c54b0380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:29.317538 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eab228-a2fb-4718-9f49-b3b3c54b0380 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:29.317818 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eab228-a2fb-4718-9f49-b3b3c54b0380 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:29.318345 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eab228-a2fb-4718-9f49-b3b3c54b0380 admin admin] PATH_INFO: `/v3/users/4780eca202394e83aec122f9a5143482` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:29.319463 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9eab228-a2fb-4718-9f49-b3b3c54b0380 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4780eca202394e83aec122f9a5143482)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:29.321397 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9eab228-a2fb-4718-9f49-b3b3c54b0380 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:29.367337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e9eab228-a2fb-4718-9f49-b3b3c54b0380 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4780eca202394e83aec122f9a5143482'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:29.367991 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e9eab228-a2fb-4718-9f49-b3b3c54b0380 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4780eca202394e83aec122f9a5143482'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:29.376310 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e9eab228-a2fb-4718-9f49-b3b3c54b0380 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4780eca202394e83aec122f9a5143482'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:29.377040 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e9eab228-a2fb-4718-9f49-b3b3c54b0380 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4780eca202394e83aec122f9a5143482'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:29.389748 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e9eab228-a2fb-4718-9f49-b3b3c54b0380 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4780eca202394e83aec122f9a5143482'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:29.401986 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1230/2501] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:08:29 2022] DELETE /identity/v3/users/4780eca202394e83aec122f9a5143482 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:29.406558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4392f7a-95a7-44f5-94a0-23028d860a99 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:29.407085 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4392f7a-95a7-44f5-94a0-23028d860a99 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:29.407198 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4392f7a-95a7-44f5-94a0-23028d860a99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:29.426468 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-d4392f7a-95a7-44f5-94a0-23028d860a99 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:08:29.427724 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d4392f7a-95a7-44f5-94a0-23028d860a99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:29.471626 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:29.490808 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1272/2502] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:08:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:08:29.500921 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9eab228-a2fb-4718-9f49-b3b3c54b0380 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:29.502349 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e9eab228-a2fb-4718-9f49-b3b3c54b0380 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:29.517546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:29.563899 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9eab228-a2fb-4718-9f49-b3b3c54b0380 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:29.567829 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ceb16b-0ea6-4125-8b90-948336807e5c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:29.568689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ceb16b-0ea6-4125-8b90-948336807e5c service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:29.568971 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ceb16b-0ea6-4125-8b90-948336807e5c service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:29.569244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ceb16b-0ea6-4125-8b90-948336807e5c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:29.579337 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ceb16b-0ea6-4125-8b90-948336807e5c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:29.580783 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ceb16b-0ea6-4125-8b90-948336807e5c service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:29.590328 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1231/2503] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:08:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:08:30.094383 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4392f7a-95a7-44f5-94a0-23028d860a99 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:30.095827 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-d4392f7a-95a7-44f5-94a0-23028d860a99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:30.109488 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4392f7a-95a7-44f5-94a0-23028d860a99 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:30.114059 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd61b1c-8ce5-43ee-9b18-67e42ac86336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:30.114692 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd61b1c-8ce5-43ee-9b18-67e42ac86336 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:30.115007 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd61b1c-8ce5-43ee-9b18-67e42ac86336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:30.115287 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd61b1c-8ce5-43ee-9b18-67e42ac86336 admin admin] PATH_INFO: `/v3/projects/34db4209796f4261aa01c1f0175e0da0` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:30.116459 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd61b1c-8ce5-43ee-9b18-67e42ac86336 admin admin] RBAC: Authorizing `identity:delete_project(project_id=34db4209796f4261aa01c1f0175e0da0)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:30.118536 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd61b1c-8ce5-43ee-9b18-67e42ac86336 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:30.168890 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-edd61b1c-8ce5-43ee-9b18-67e42ac86336 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '34db4209796f4261aa01c1f0175e0da0', 'request_id': 'req-edd61b1c-8ce5-43ee-9b18-67e42ac86336'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:30.175927 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-edd61b1c-8ce5-43ee-9b18-67e42ac86336 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '34db4209796f4261aa01c1f0175e0da0', 'request_id': 'req-edd61b1c-8ce5-43ee-9b18-67e42ac86336'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:30.184338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-edd61b1c-8ce5-43ee-9b18-67e42ac86336 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '34db4209796f4261aa01c1f0175e0da0', 'request_id': 'req-edd61b1c-8ce5-43ee-9b18-67e42ac86336'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:30.196097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-edd61b1c-8ce5-43ee-9b18-67e42ac86336 admin admin] The token cache is being invalidate because project 34db4209796f4261aa01c1f0175e0da0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:08:30.196386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-edd61b1c-8ce5-43ee-9b18-67e42ac86336 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:30.197317 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-edd61b1c-8ce5-43ee-9b18-67e42ac86336 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:30.198903 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1273/2504] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:08:30 2022] DELETE /identity/v3/projects/34db4209796f4261aa01c1f0175e0da0 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:30.203762 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6fe9f2-fb2c-4dbd-a5ac-4905b0a847c4 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:30.204043 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6fe9f2-fb2c-4dbd-a5ac-4905b0a847c4 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:30.204341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6fe9f2-fb2c-4dbd-a5ac-4905b0a847c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:30.224393 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-be6fe9f2-fb2c-4dbd-a5ac-4905b0a847c4 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:08:30.225613 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-be6fe9f2-fb2c-4dbd-a5ac-4905b0a847c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:30.286066 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1232/2505] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:08:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 01:08:30.295905 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0553322-ae75-475c-9971-74c6e0975811 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:30.296181 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0553322-ae75-475c-9971-74c6e0975811 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:30.296464 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0553322-ae75-475c-9971-74c6e0975811 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:30.316455 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-a0553322-ae75-475c-9971-74c6e0975811 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:08:30.317667 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-a0553322-ae75-475c-9971-74c6e0975811 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:30.325790 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1274/2506] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:08:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:08:30.330017 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6fe9f2-fb2c-4dbd-a5ac-4905b0a847c4 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:30.340306 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6fe9f2-fb2c-4dbd-a5ac-4905b0a847c4 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:30.344889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471272f0-9fb6-4763-a14b-ea4540ba4575 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:30.345488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471272f0-9fb6-4763-a14b-ea4540ba4575 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:30.345798 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471272f0-9fb6-4763-a14b-ea4540ba4575 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:30.346072 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471272f0-9fb6-4763-a14b-ea4540ba4575 admin admin] PATH_INFO: `/v3/domains` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:30.346743 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471272f0-9fb6-4763-a14b-ea4540ba4575 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81485) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:08:30.347078 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471272f0-9fb6-4763-a14b-ea4540ba4575 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:30.348689 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471272f0-9fb6-4763-a14b-ea4540ba4575 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:30.360831 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1233/2507] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:08:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:08:30.365185 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0553322-ae75-475c-9971-74c6e0975811 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:30.374661 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0553322-ae75-475c-9971-74c6e0975811 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:30.379330 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc90f79d-1a3d-410e-a5a4-667ca8b23728 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:30.380068 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc90f79d-1a3d-410e-a5a4-667ca8b23728 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:30.380359 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc90f79d-1a3d-410e-a5a4-667ca8b23728 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:30.380635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc90f79d-1a3d-410e-a5a4-667ca8b23728 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:30.381270 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc90f79d-1a3d-410e-a5a4-667ca8b23728 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:30.383211 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc90f79d-1a3d-410e-a5a4-667ca8b23728 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:30.411176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1275/2508] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:08:30 2022] POST /identity/v3/projects => generated 405 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 16 01:08:30.415276 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471272f0-9fb6-4763-a14b-ea4540ba4575 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:30.425993 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471272f0-9fb6-4763-a14b-ea4540ba4575 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:30.481700 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b8aacb-cc1c-470a-b17f-dd605fb4f55d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:30.482308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b8aacb-cc1c-470a-b17f-dd605fb4f55d admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:30.482605 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b8aacb-cc1c-470a-b17f-dd605fb4f55d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:30.482888 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b8aacb-cc1c-470a-b17f-dd605fb4f55d admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:30.483520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b8aacb-cc1c-470a-b17f-dd605fb4f55d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:30.485443 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b8aacb-cc1c-470a-b17f-dd605fb4f55d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:30.506108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1234/2509] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:08:30 2022] POST /identity/v3/users => generated 303 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:08:30.511016 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc90f79d-1a3d-410e-a5a4-667ca8b23728 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:30.521267 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc90f79d-1a3d-410e-a5a4-667ca8b23728 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:30.526590 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f69eb7-b818-4c3e-83f9-ed0e8ec23659 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:30.527228 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f69eb7-b818-4c3e-83f9-ed0e8ec23659 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:30.527514 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f69eb7-b818-4c3e-83f9-ed0e8ec23659 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:30.527838 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f69eb7-b818-4c3e-83f9-ed0e8ec23659 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:30.528487 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f69eb7-b818-4c3e-83f9-ed0e8ec23659 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:30.530092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f69eb7-b818-4c3e-83f9-ed0e8ec23659 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:30.540036 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1276/2510] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:08:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 01:08:30.544120 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b8aacb-cc1c-470a-b17f-dd605fb4f55d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:30.553887 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b8aacb-cc1c-470a-b17f-dd605fb4f55d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:30.558763 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52a2e80-3223-44b8-925e-c32b3b347b7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:30.559387 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52a2e80-3223-44b8-925e-c32b3b347b7e admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:30.559671 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52a2e80-3223-44b8-925e-c32b3b347b7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:30.559942 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52a2e80-3223-44b8-925e-c32b3b347b7e admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:30.560606 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52a2e80-3223-44b8-925e-c32b3b347b7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:30.562268 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52a2e80-3223-44b8-925e-c32b3b347b7e admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:30.573848 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1235/2511] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:08:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:08:30.578132 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f69eb7-b818-4c3e-83f9-ed0e8ec23659 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:30.588577 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f69eb7-b818-4c3e-83f9-ed0e8ec23659 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:30.593210 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a22a8e-1e6b-4817-ab7a-3285a9c905f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:30.593855 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a22a8e-1e6b-4817-ab7a-3285a9c905f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:30.594132 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a22a8e-1e6b-4817-ab7a-3285a9c905f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:30.594404 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a22a8e-1e6b-4817-ab7a-3285a9c905f4 admin admin] PATH_INFO: `/v3/projects/d9680903f6f54428821707737e370139/users/ea157b90dc6547a6940188ee9b6bea45/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:30.609212 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a22a8e-1e6b-4817-ab7a-3285a9c905f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d9680903f6f54428821707737e370139, user_id=ea157b90dc6547a6940188ee9b6bea45, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:30.611914 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a22a8e-1e6b-4817-ab7a-3285a9c905f4 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:30.624746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1277/2512] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:08:30 2022] PUT /identity/v3/projects/d9680903f6f54428821707737e370139/users/ea157b90dc6547a6940188ee9b6bea45/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:30.636768 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c0ef2f-7fbf-40e3-a0ce-298570ed117e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:30.637056 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c0ef2f-7fbf-40e3-a0ce-298570ed117e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:30.637330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7c0ef2f-7fbf-40e3-a0ce-298570ed117e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:30.665549 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b7c0ef2f-7fbf-40e3-a0ce-298570ed117e None None] MFA Rules not processed for user `ea157b90dc6547a6940188ee9b6bea45`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:08:30.666717 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b7c0ef2f-7fbf-40e3-a0ce-298570ed117e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:30.676330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:30.735698 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1236/2513] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:08:30 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 16 01:08:30.740689 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a22a8e-1e6b-4817-ab7a-3285a9c905f4 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:30.803215 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a22a8e-1e6b-4817-ab7a-3285a9c905f4 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:30.808107 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2acfb88-0532-45b7-8bca-a330cf22dc44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:30.808712 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2acfb88-0532-45b7-8bca-a330cf22dc44 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:30.808997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2acfb88-0532-45b7-8bca-a330cf22dc44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:30.809264 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2acfb88-0532-45b7-8bca-a330cf22dc44 admin admin] PATH_INFO: `/v3/projects` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:30.809896 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2acfb88-0532-45b7-8bca-a330cf22dc44 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:30.811834 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2acfb88-0532-45b7-8bca-a330cf22dc44 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:30.823508 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:30.841032 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1278/2514] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:08:30 2022] POST /identity/v3/projects => generated 403 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:08:30.845231 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c0ef2f-7fbf-40e3-a0ce-298570ed117e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:30.855828 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7c0ef2f-7fbf-40e3-a0ce-298570ed117e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:30.912256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf40f3a-7bdf-4842-accf-a1ed68aa847c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:30.912867 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf40f3a-7bdf-4842-accf-a1ed68aa847c admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:30.913147 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf40f3a-7bdf-4842-accf-a1ed68aa847c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:30.913421 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf40f3a-7bdf-4842-accf-a1ed68aa847c admin admin] PATH_INFO: `/v3/users` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:30.914052 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf40f3a-7bdf-4842-accf-a1ed68aa847c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:30.916081 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf40f3a-7bdf-4842-accf-a1ed68aa847c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:30.938444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1237/2515] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:08:30 2022] POST /identity/v3/users => generated 308 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:08:30.942386 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2acfb88-0532-45b7-8bca-a330cf22dc44 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:30.953600 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2acfb88-0532-45b7-8bca-a330cf22dc44 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:30.958276 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d4ff57-f3f1-4fa1-bf0c-07599a2665f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:30.958891 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d4ff57-f3f1-4fa1-bf0c-07599a2665f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:30.959181 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d4ff57-f3f1-4fa1-bf0c-07599a2665f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:30.959455 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d4ff57-f3f1-4fa1-bf0c-07599a2665f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:30.960096 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d4ff57-f3f1-4fa1-bf0c-07599a2665f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:30.961691 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d4ff57-f3f1-4fa1-bf0c-07599a2665f0 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:30.971733 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1279/2516] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:08:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:08:30.976246 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf40f3a-7bdf-4842-accf-a1ed68aa847c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:30.986542 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf40f3a-7bdf-4842-accf-a1ed68aa847c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:30.991293 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b360e3b-94e4-46e6-93d9-6fa082de2104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:30.991948 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b360e3b-94e4-46e6-93d9-6fa082de2104 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:30.992232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b360e3b-94e4-46e6-93d9-6fa082de2104 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:30.992507 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b360e3b-94e4-46e6-93d9-6fa082de2104 admin admin] PATH_INFO: `/v3/projects/f4395f4aa2f1419c8eab0a48979597eb/users/5ff57198a5454cc9bee0cadc9fde4c2a/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:31.008552 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b360e3b-94e4-46e6-93d9-6fa082de2104 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f4395f4aa2f1419c8eab0a48979597eb, user_id=5ff57198a5454cc9bee0cadc9fde4c2a, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:31.012160 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b360e3b-94e4-46e6-93d9-6fa082de2104 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:31.028024 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1238/2517] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:08:30 2022] PUT /identity/v3/projects/f4395f4aa2f1419c8eab0a48979597eb/users/5ff57198a5454cc9bee0cadc9fde4c2a/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:31.032480 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d4ff57-f3f1-4fa1-bf0c-07599a2665f0 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:31.044176 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d4ff57-f3f1-4fa1-bf0c-07599a2665f0 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:31.102787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3988981d-f9ca-40d6-bb98-b7046a828060 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:31.103432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3988981d-f9ca-40d6-bb98-b7046a828060 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:31.103717 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3988981d-f9ca-40d6-bb98-b7046a828060 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:31.103993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3988981d-f9ca-40d6-bb98-b7046a828060 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:31.104644 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3988981d-f9ca-40d6-bb98-b7046a828060 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:31.106241 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3988981d-f9ca-40d6-bb98-b7046a828060 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:31.116028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1280/2518] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:08:31 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:08:31.121065 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b360e3b-94e4-46e6-93d9-6fa082de2104 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:31.131481 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b360e3b-94e4-46e6-93d9-6fa082de2104 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:31.136824 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1260fb4-a11a-4bab-bb39-49ab5e5880f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:31.137465 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1260fb4-a11a-4bab-bb39-49ab5e5880f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:31.137742 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1260fb4-a11a-4bab-bb39-49ab5e5880f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:31.138013 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1260fb4-a11a-4bab-bb39-49ab5e5880f9 admin admin] PATH_INFO: `/v3/projects/f4395f4aa2f1419c8eab0a48979597eb/users/5ff57198a5454cc9bee0cadc9fde4c2a/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:31.140004 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1260fb4-a11a-4bab-bb39-49ab5e5880f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f4395f4aa2f1419c8eab0a48979597eb, user_id=5ff57198a5454cc9bee0cadc9fde4c2a, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:31.142711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1260fb4-a11a-4bab-bb39-49ab5e5880f9 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:31.153065 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1239/2519] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:08:31 2022] PUT /identity/v3/projects/f4395f4aa2f1419c8eab0a48979597eb/users/5ff57198a5454cc9bee0cadc9fde4c2a/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:31.164531 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d19b30a-d1af-4c6c-9dfc-d29aef07c601 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:31.164826 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d19b30a-d1af-4c6c-9dfc-d29aef07c601 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:31.165347 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d19b30a-d1af-4c6c-9dfc-d29aef07c601 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:31.199401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-0d19b30a-d1af-4c6c-9dfc-d29aef07c601 None None] MFA Rules not processed for user `5ff57198a5454cc9bee0cadc9fde4c2a`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:08:31.200532 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-0d19b30a-d1af-4c6c-9dfc-d29aef07c601 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:31.268804 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1281/2520] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:08:31 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:08:31.280647 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1260fb4-a11a-4bab-bb39-49ab5e5880f9 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:31.282466 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e1260fb4-a11a-4bab-bb39-49ab5e5880f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:31.340354 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1260fb4-a11a-4bab-bb39-49ab5e5880f9 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:31.344158 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9751b510-ac90-4b39-8c3e-43f044bbed6e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:31.344765 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9751b510-ac90-4b39-8c3e-43f044bbed6e service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:31.345049 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9751b510-ac90-4b39-8c3e-43f044bbed6e service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:31.345327 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9751b510-ac90-4b39-8c3e-43f044bbed6e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:31.354794 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9751b510-ac90-4b39-8c3e-43f044bbed6e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:31.356268 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9751b510-ac90-4b39-8c3e-43f044bbed6e service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:31.365881 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1240/2521] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:08:31 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:08:31.509432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d19b30a-d1af-4c6c-9dfc-d29aef07c601 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:31.573624 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d19b30a-d1af-4c6c-9dfc-d29aef07c601 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:31.578273 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0043418-5803-4996-ae88-2c191c8892ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:31.578934 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0043418-5803-4996-ae88-2c191c8892ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:31.579221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0043418-5803-4996-ae88-2c191c8892ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:31.579501 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0043418-5803-4996-ae88-2c191c8892ea admin admin] PATH_INFO: `/v3/users/ea157b90dc6547a6940188ee9b6bea45` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:31.580612 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0043418-5803-4996-ae88-2c191c8892ea admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea157b90dc6547a6940188ee9b6bea45)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:31.582623 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0043418-5803-4996-ae88-2c191c8892ea admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:31.585069 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:31.630552 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b0043418-5803-4996-ae88-2c191c8892ea admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea157b90dc6547a6940188ee9b6bea45'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:31.631220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b0043418-5803-4996-ae88-2c191c8892ea admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea157b90dc6547a6940188ee9b6bea45'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:31.639479 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b0043418-5803-4996-ae88-2c191c8892ea admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea157b90dc6547a6940188ee9b6bea45'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:31.651560 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b0043418-5803-4996-ae88-2c191c8892ea admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea157b90dc6547a6940188ee9b6bea45'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:31.658331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b0043418-5803-4996-ae88-2c191c8892ea admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea157b90dc6547a6940188ee9b6bea45'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:31.664335 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1282/2522] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:08:31 2022] DELETE /identity/v3/users/ea157b90dc6547a6940188ee9b6bea45 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:31.669126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a61a85-1684-4994-812f-e6599ab51459 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:31.669368 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a61a85-1684-4994-812f-e6599ab51459 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:31.669639 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a61a85-1684-4994-812f-e6599ab51459 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:31.683763 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:31.690308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-c1a61a85-1684-4994-812f-e6599ab51459 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:08:31.691532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c1a61a85-1684-4994-812f-e6599ab51459 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:31.752386 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1241/2523] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:08:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:08:31.762574 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0043418-5803-4996-ae88-2c191c8892ea admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:31.764097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b0043418-5803-4996-ae88-2c191c8892ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:31.822212 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0043418-5803-4996-ae88-2c191c8892ea admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:31.825916 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45f8176-2188-4bae-b4c7-af21faf8c1e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:31.826525 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45f8176-2188-4bae-b4c7-af21faf8c1e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:31.826809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45f8176-2188-4bae-b4c7-af21faf8c1e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:31.827121 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45f8176-2188-4bae-b4c7-af21faf8c1e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:31.836203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45f8176-2188-4bae-b4c7-af21faf8c1e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:31.838179 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45f8176-2188-4bae-b4c7-af21faf8c1e8 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:31.847903 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1283/2524] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:08:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:08:32.116371 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a61a85-1684-4994-812f-e6599ab51459 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:32.117918 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-c1a61a85-1684-4994-812f-e6599ab51459 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:32.136552 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a61a85-1684-4994-812f-e6599ab51459 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:32.141374 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:32.143149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:32.143149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:32.143542 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12 admin admin] PATH_INFO: `/v3/projects/d9680903f6f54428821707737e370139` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:32.144537 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d9680903f6f54428821707737e370139)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:32.146537 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:32.195462 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd9680903f6f54428821707737e370139', 'request_id': 'req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:32.204617 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd9680903f6f54428821707737e370139', 'request_id': 'req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:32.211382 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd9680903f6f54428821707737e370139', 'request_id': 'req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:32.221927 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12 admin admin] The token cache is being invalidate because project d9680903f6f54428821707737e370139 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:08:32.222210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:32.222883 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:32.224368 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1242/2525] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:08:32 2022] DELETE /identity/v3/projects/d9680903f6f54428821707737e370139 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:32.228911 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45f8176-2188-4bae-b4c7-af21faf8c1e8 service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:32.238933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45f8176-2188-4bae-b4c7-af21faf8c1e8 service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:32.297406 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675951f0-53d4-44df-bc9a-4fa9df966486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:32.298048 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675951f0-53d4-44df-bc9a-4fa9df966486 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:32.298327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675951f0-53d4-44df-bc9a-4fa9df966486 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:32.298600 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675951f0-53d4-44df-bc9a-4fa9df966486 admin admin] PATH_INFO: `/v3/users/5ff57198a5454cc9bee0cadc9fde4c2a` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:32.299778 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675951f0-53d4-44df-bc9a-4fa9df966486 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5ff57198a5454cc9bee0cadc9fde4c2a)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:32.301705 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675951f0-53d4-44df-bc9a-4fa9df966486 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:32.330959 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:32.351097 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-675951f0-53d4-44df-bc9a-4fa9df966486 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5ff57198a5454cc9bee0cadc9fde4c2a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:32.351750 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-675951f0-53d4-44df-bc9a-4fa9df966486 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5ff57198a5454cc9bee0cadc9fde4c2a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:32.359132 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-675951f0-53d4-44df-bc9a-4fa9df966486 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5ff57198a5454cc9bee0cadc9fde4c2a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:32.372148 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-675951f0-53d4-44df-bc9a-4fa9df966486 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5ff57198a5454cc9bee0cadc9fde4c2a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:32.379119 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-675951f0-53d4-44df-bc9a-4fa9df966486 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5ff57198a5454cc9bee0cadc9fde4c2a'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:32.385373 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1284/2526] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:08:32 2022] DELETE /identity/v3/users/5ff57198a5454cc9bee0cadc9fde4c2a => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:32.658174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:32.659727 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:32.673508 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def0df3f-0ca0-4ba9-8a9d-06477ede6f12 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:32.732225 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a3f9a3-a111-4463-bd7e-e833c59f92ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:32.732861 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a3f9a3-a111-4463-bd7e-e833c59f92ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:32.733166 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a3f9a3-a111-4463-bd7e-e833c59f92ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:32.733498 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a3f9a3-a111-4463-bd7e-e833c59f92ca admin admin] PATH_INFO: `/v3/projects/f4395f4aa2f1419c8eab0a48979597eb` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:32.734606 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a3f9a3-a111-4463-bd7e-e833c59f92ca admin admin] RBAC: Authorizing `identity:delete_project(project_id=f4395f4aa2f1419c8eab0a48979597eb)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:32.736580 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a3f9a3-a111-4463-bd7e-e833c59f92ca admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:32.771822 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:32.785748 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b4a3f9a3-a111-4463-bd7e-e833c59f92ca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f4395f4aa2f1419c8eab0a48979597eb', 'request_id': 'req-b4a3f9a3-a111-4463-bd7e-e833c59f92ca'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:32.792998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b4a3f9a3-a111-4463-bd7e-e833c59f92ca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f4395f4aa2f1419c8eab0a48979597eb', 'request_id': 'req-b4a3f9a3-a111-4463-bd7e-e833c59f92ca'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:32.799669 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b4a3f9a3-a111-4463-bd7e-e833c59f92ca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f4395f4aa2f1419c8eab0a48979597eb', 'request_id': 'req-b4a3f9a3-a111-4463-bd7e-e833c59f92ca'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:32.810096 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b4a3f9a3-a111-4463-bd7e-e833c59f92ca admin admin] The token cache is being invalidate because project f4395f4aa2f1419c8eab0a48979597eb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:08:32.810318 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b4a3f9a3-a111-4463-bd7e-e833c59f92ca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:32.811005 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-b4a3f9a3-a111-4463-bd7e-e833c59f92ca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:08:32.812481 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1243/2527] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:08:32 2022] DELETE /identity/v3/projects/f4395f4aa2f1419c8eab0a48979597eb => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:32.817311 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f8d687-03e0-424b-93bf-52c71cca8f35 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:32.817805 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f8d687-03e0-424b-93bf-52c71cca8f35 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:32.817913 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f8d687-03e0-424b-93bf-52c71cca8f35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:32.838724 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-54f8d687-03e0-424b-93bf-52c71cca8f35 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:08:32.839943 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-54f8d687-03e0-424b-93bf-52c71cca8f35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:32.898362 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1285/2528] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:08:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:08:32.906686 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0463239d-98c7-47c4-96f1-96a30fda80e8 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:32.907008 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0463239d-98c7-47c4-96f1-96a30fda80e8 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:32.907288 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0463239d-98c7-47c4-96f1-96a30fda80e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:32.927588 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-0463239d-98c7-47c4-96f1-96a30fda80e8 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:08:32.928821 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-0463239d-98c7-47c4-96f1-96a30fda80e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:32.938927 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1244/2529] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:08:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:08:32.943685 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f8d687-03e0-424b-93bf-52c71cca8f35 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:32.954074 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f8d687-03e0-424b-93bf-52c71cca8f35 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:32.958638 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ee459c-b598-4c7b-941f-fbdd0f74b9c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:32.959272 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ee459c-b598-4c7b-941f-fbdd0f74b9c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:32.959546 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ee459c-b598-4c7b-941f-fbdd0f74b9c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:32.959813 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ee459c-b598-4c7b-941f-fbdd0f74b9c5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:32.960452 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ee459c-b598-4c7b-941f-fbdd0f74b9c5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:08:32.960751 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ee459c-b598-4c7b-941f-fbdd0f74b9c5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:32.962981 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ee459c-b598-4c7b-941f-fbdd0f74b9c5 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:32.974493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1286/2530] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:08:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 01:08:32.979095 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0463239d-98c7-47c4-96f1-96a30fda80e8 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:32.988446 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0463239d-98c7-47c4-96f1-96a30fda80e8 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:32.993207 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def94bbf-c4ea-4763-9823-69a08c5eba76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:32.993919 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def94bbf-c4ea-4763-9823-69a08c5eba76 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:32.994195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def94bbf-c4ea-4763-9823-69a08c5eba76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:32.994472 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def94bbf-c4ea-4763-9823-69a08c5eba76 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:32.995148 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def94bbf-c4ea-4763-9823-69a08c5eba76 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:32.997084 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def94bbf-c4ea-4763-9823-69a08c5eba76 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:33.025788 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1245/2531] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:08:32 2022] POST /identity/v3/projects => generated 377 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:08:33.030374 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ee459c-b598-4c7b-941f-fbdd0f74b9c5 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:33.040845 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ee459c-b598-4c7b-941f-fbdd0f74b9c5 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:33.098601 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33aec0b8-6eec-431f-9ccc-114cbf0dce90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:33.099164 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33aec0b8-6eec-431f-9ccc-114cbf0dce90 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:33.099445 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33aec0b8-6eec-431f-9ccc-114cbf0dce90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:33.099713 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33aec0b8-6eec-431f-9ccc-114cbf0dce90 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:33.100334 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33aec0b8-6eec-431f-9ccc-114cbf0dce90 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:33.102242 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33aec0b8-6eec-431f-9ccc-114cbf0dce90 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:33.124036 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1287/2532] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:08:33 2022] POST /identity/v3/users => generated 289 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:08:33.128161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def94bbf-c4ea-4763-9823-69a08c5eba76 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:33.139690 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def94bbf-c4ea-4763-9823-69a08c5eba76 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:33.144771 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac108225-9d43-4eb6-b764-23a16f460af3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:33.145406 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac108225-9d43-4eb6-b764-23a16f460af3 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:33.145690 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac108225-9d43-4eb6-b764-23a16f460af3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:33.145969 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac108225-9d43-4eb6-b764-23a16f460af3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:33.146612 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac108225-9d43-4eb6-b764-23a16f460af3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:33.148254 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac108225-9d43-4eb6-b764-23a16f460af3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:33.158287 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1246/2533] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:08:33 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:08:33.162256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33aec0b8-6eec-431f-9ccc-114cbf0dce90 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:33.172286 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33aec0b8-6eec-431f-9ccc-114cbf0dce90 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:33.176829 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be927414-d41f-4dd9-aa32-a039fb000c24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:33.177424 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be927414-d41f-4dd9-aa32-a039fb000c24 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:33.177701 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be927414-d41f-4dd9-aa32-a039fb000c24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:33.178092 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be927414-d41f-4dd9-aa32-a039fb000c24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:33.178628 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be927414-d41f-4dd9-aa32-a039fb000c24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:33.180502 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be927414-d41f-4dd9-aa32-a039fb000c24 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:33.192011 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1288/2534] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:08:33 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:08:33.197310 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac108225-9d43-4eb6-b764-23a16f460af3 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:33.208071 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac108225-9d43-4eb6-b764-23a16f460af3 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:33.212856 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2ec6e9-66e8-4635-bfac-ddcf44553908 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:33.213575 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2ec6e9-66e8-4635-bfac-ddcf44553908 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:33.213802 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2ec6e9-66e8-4635-bfac-ddcf44553908 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:33.214077 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2ec6e9-66e8-4635-bfac-ddcf44553908 admin admin] PATH_INFO: `/v3/projects/35a285fc9170459395ef7304b4c468d6/users/f513d4177bc147f18ba70d1454435e94/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:33.228971 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e2ec6e9-66e8-4635-bfac-ddcf44553908 admin admin] RBAC: Authorizing `identity:create_grant(project_id=35a285fc9170459395ef7304b4c468d6, user_id=f513d4177bc147f18ba70d1454435e94, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:33.231718 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e2ec6e9-66e8-4635-bfac-ddcf44553908 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:33.245348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1247/2535] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:08:33 2022] PUT /identity/v3/projects/35a285fc9170459395ef7304b4c468d6/users/f513d4177bc147f18ba70d1454435e94/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:33.256074 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c662ed4-fdff-4fc9-a480-b20fa349f3a7 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:33.256364 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c662ed4-fdff-4fc9-a480-b20fa349f3a7 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:33.256639 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c662ed4-fdff-4fc9-a480-b20fa349f3a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:33.277045 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:33.287229 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-3c662ed4-fdff-4fc9-a480-b20fa349f3a7 None None] MFA Rules not processed for user `f513d4177bc147f18ba70d1454435e94`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:08:33.288704 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-3c662ed4-fdff-4fc9-a480-b20fa349f3a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:33.352797 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1289/2536] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:08:33 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:08:33.357393 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2ec6e9-66e8-4635-bfac-ddcf44553908 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:33.419032 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2ec6e9-66e8-4635-bfac-ddcf44553908 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:33.423807 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fa94fd-935d-4511-9ad7-2a591f58b569 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:33.424424 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fa94fd-935d-4511-9ad7-2a591f58b569 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:33.424703 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fa94fd-935d-4511-9ad7-2a591f58b569 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:33.424976 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fa94fd-935d-4511-9ad7-2a591f58b569 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:33.425602 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fa94fd-935d-4511-9ad7-2a591f58b569 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:33.427604 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fa94fd-935d-4511-9ad7-2a591f58b569 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:33.457362 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1248/2537] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:08:33 2022] POST /identity/v3/projects => generated 377 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:08:33.461522 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c662ed4-fdff-4fc9-a480-b20fa349f3a7 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:33.472719 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c662ed4-fdff-4fc9-a480-b20fa349f3a7 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:33.532510 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190a55c0-4a09-4a8d-af7b-9be84e07190c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:33.533131 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190a55c0-4a09-4a8d-af7b-9be84e07190c admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:33.533415 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190a55c0-4a09-4a8d-af7b-9be84e07190c admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:33.533691 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190a55c0-4a09-4a8d-af7b-9be84e07190c admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:33.534340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190a55c0-4a09-4a8d-af7b-9be84e07190c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:33.536303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190a55c0-4a09-4a8d-af7b-9be84e07190c admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:33.556877 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1290/2538] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:08:33 2022] POST /identity/v3/users => generated 295 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:08:33.560951 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fa94fd-935d-4511-9ad7-2a591f58b569 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:33.572025 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fa94fd-935d-4511-9ad7-2a591f58b569 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:33.577066 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9810375-86ea-44f8-9b9d-43cf1ab503aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:33.577719 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9810375-86ea-44f8-9b9d-43cf1ab503aa admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:33.578121 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9810375-86ea-44f8-9b9d-43cf1ab503aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:33.578444 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9810375-86ea-44f8-9b9d-43cf1ab503aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:33.579216 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9810375-86ea-44f8-9b9d-43cf1ab503aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:33.580902 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9810375-86ea-44f8-9b9d-43cf1ab503aa admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:33.592149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1249/2539] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:08:33 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:08:33.596327 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190a55c0-4a09-4a8d-af7b-9be84e07190c admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:33.606439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190a55c0-4a09-4a8d-af7b-9be84e07190c admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:33.611014 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b72cad-728a-40f4-a46f-f7a2a7fd3bb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:33.611677 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b72cad-728a-40f4-a46f-f7a2a7fd3bb8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:33.611956 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b72cad-728a-40f4-a46f-f7a2a7fd3bb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:33.612228 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b72cad-728a-40f4-a46f-f7a2a7fd3bb8 admin admin] PATH_INFO: `/v3/projects/413fd1eccc34426cbf4341a2f0faa239/users/09f09b11a81842308dde5186f5754138/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:33.628089 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b72cad-728a-40f4-a46f-f7a2a7fd3bb8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=413fd1eccc34426cbf4341a2f0faa239, user_id=09f09b11a81842308dde5186f5754138, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:33.630787 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b72cad-728a-40f4-a46f-f7a2a7fd3bb8 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:33.644527 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1291/2540] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:08:33 2022] PUT /identity/v3/projects/413fd1eccc34426cbf4341a2f0faa239/users/09f09b11a81842308dde5186f5754138/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:33.648532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9810375-86ea-44f8-9b9d-43cf1ab503aa admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:33.659161 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9810375-86ea-44f8-9b9d-43cf1ab503aa admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:33.689191 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:33.719308 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bde4bae-7afd-4d92-8eb5-dc86b6dbfe51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:33.719914 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bde4bae-7afd-4d92-8eb5-dc86b6dbfe51 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:33.720195 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bde4bae-7afd-4d92-8eb5-dc86b6dbfe51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:33.720466 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bde4bae-7afd-4d92-8eb5-dc86b6dbfe51 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:33.721096 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bde4bae-7afd-4d92-8eb5-dc86b6dbfe51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:33.722687 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bde4bae-7afd-4d92-8eb5-dc86b6dbfe51 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:33.732952 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1250/2541] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:08:33 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:08:33.737161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b72cad-728a-40f4-a46f-f7a2a7fd3bb8 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:33.748109 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b72cad-728a-40f4-a46f-f7a2a7fd3bb8 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:33.752627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085162ed-16ff-43dd-95f4-81ac2a3025a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:33.753380 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085162ed-16ff-43dd-95f4-81ac2a3025a2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:33.753555 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085162ed-16ff-43dd-95f4-81ac2a3025a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:33.753826 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085162ed-16ff-43dd-95f4-81ac2a3025a2 admin admin] PATH_INFO: `/v3/projects/413fd1eccc34426cbf4341a2f0faa239/users/09f09b11a81842308dde5186f5754138/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:33.755748 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085162ed-16ff-43dd-95f4-81ac2a3025a2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=413fd1eccc34426cbf4341a2f0faa239, user_id=09f09b11a81842308dde5186f5754138, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:33.758378 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085162ed-16ff-43dd-95f4-81ac2a3025a2 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:33.768848 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1292/2542] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:08:33 2022] PUT /identity/v3/projects/413fd1eccc34426cbf4341a2f0faa239/users/09f09b11a81842308dde5186f5754138/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:08:33.834193 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4490389-2271-4aa4-9ac0-1cb32266b343 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:33.834482 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4490389-2271-4aa4-9ac0-1cb32266b343 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:33.834765 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4490389-2271-4aa4-9ac0-1cb32266b343 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:33.863107 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b4490389-2271-4aa4-9ac0-1cb32266b343 None None] MFA Rules not processed for user `09f09b11a81842308dde5186f5754138`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:08:33.864258 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b4490389-2271-4aa4-9ac0-1cb32266b343 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:33.932864 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1251/2543] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:08:33 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:08:33.943995 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085162ed-16ff-43dd-95f4-81ac2a3025a2 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:33.945745 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-085162ed-16ff-43dd-95f4-81ac2a3025a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:08:34.003190 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085162ed-16ff-43dd-95f4-81ac2a3025a2 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:34.008214 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d316cd-6e42-425a-b0ad-696679161cba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:34.008826 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d316cd-6e42-425a-b0ad-696679161cba service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:34.009127 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d316cd-6e42-425a-b0ad-696679161cba service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:34.009397 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d316cd-6e42-425a-b0ad-696679161cba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:34.013985 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:34.072558 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d316cd-6e42-425a-b0ad-696679161cba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:34.073997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d316cd-6e42-425a-b0ad-696679161cba service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:34.083007 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1293/2544] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:08:33 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:08:38.718400 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4490389-2271-4aa4-9ac0-1cb32266b343 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:08:38.728871 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4490389-2271-4aa4-9ac0-1cb32266b343 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:08:38.732495 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725602be-74d7-4bc0-b8c5-07882d36d58f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:08:38.733131 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725602be-74d7-4bc0-b8c5-07882d36d58f service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:08:38.733413 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725602be-74d7-4bc0-b8c5-07882d36d58f service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:08:38.734232 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725602be-74d7-4bc0-b8c5-07882d36d58f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:08:38.743248 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725602be-74d7-4bc0-b8c5-07882d36d58f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:08:38.744706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725602be-74d7-4bc0-b8c5-07882d36d58f service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:08:38.754454 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1252/2545] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:08:38 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:08:44.078878 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=25 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:08:48.750098 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=55 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:02.619927 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d316cd-6e42-425a-b0ad-696679161cba service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:02.682745 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d316cd-6e42-425a-b0ad-696679161cba service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:02.687899 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90394e12-6b62-4d16-b6e5-abe60017731a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:02.688533 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90394e12-6b62-4d16-b6e5-abe60017731a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:02.688816 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90394e12-6b62-4d16-b6e5-abe60017731a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:02.689093 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90394e12-6b62-4d16-b6e5-abe60017731a admin admin] PATH_INFO: `/v3/users/f513d4177bc147f18ba70d1454435e94` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:02.690208 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90394e12-6b62-4d16-b6e5-abe60017731a admin admin] RBAC: Authorizing `identity:delete_user(user_id=f513d4177bc147f18ba70d1454435e94)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:02.692181 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90394e12-6b62-4d16-b6e5-abe60017731a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:02.738189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-90394e12-6b62-4d16-b6e5-abe60017731a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f513d4177bc147f18ba70d1454435e94'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:02.738856 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-90394e12-6b62-4d16-b6e5-abe60017731a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f513d4177bc147f18ba70d1454435e94'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:02.747641 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-90394e12-6b62-4d16-b6e5-abe60017731a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f513d4177bc147f18ba70d1454435e94'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:02.759790 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-90394e12-6b62-4d16-b6e5-abe60017731a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f513d4177bc147f18ba70d1454435e94'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:02.767038 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-90394e12-6b62-4d16-b6e5-abe60017731a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f513d4177bc147f18ba70d1454435e94'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:02.772937 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1294/2546] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:09:02 2022] DELETE /identity/v3/users/f513d4177bc147f18ba70d1454435e94 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:09:02.777441 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b158cfe-5b92-434a-9512-4080b93faebb None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:02.778142 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b158cfe-5b92-434a-9512-4080b93faebb None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:02.778440 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b158cfe-5b92-434a-9512-4080b93faebb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:02.798773 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2b158cfe-5b92-434a-9512-4080b93faebb None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:09:02.800095 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2b158cfe-5b92-434a-9512-4080b93faebb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:02.859590 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1253/2547] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:09:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:09:02.869188 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90394e12-6b62-4d16-b6e5-abe60017731a admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:02.870611 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-90394e12-6b62-4d16-b6e5-abe60017731a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:02.929338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90394e12-6b62-4d16-b6e5-abe60017731a admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:02.933088 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44040dd7-57ea-4e78-b8ab-a40e40b0a446 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:02.934354 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44040dd7-57ea-4e78-b8ab-a40e40b0a446 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:02.934652 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44040dd7-57ea-4e78-b8ab-a40e40b0a446 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:02.935116 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44040dd7-57ea-4e78-b8ab-a40e40b0a446 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:02.945370 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44040dd7-57ea-4e78-b8ab-a40e40b0a446 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:02.946800 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44040dd7-57ea-4e78-b8ab-a40e40b0a446 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:02.956556 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1295/2548] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:09:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:09:03.235022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b158cfe-5b92-434a-9512-4080b93faebb None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:03.236500 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2b158cfe-5b92-434a-9512-4080b93faebb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:03.248836 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b158cfe-5b92-434a-9512-4080b93faebb None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:03.253434 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:03.254070 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:03.254350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:03.254618 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4 admin admin] PATH_INFO: `/v3/projects/35a285fc9170459395ef7304b4c468d6` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:03.255760 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=35a285fc9170459395ef7304b4c468d6)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:03.257674 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:03.305049 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '35a285fc9170459395ef7304b4c468d6', 'request_id': 'req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:03.312821 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '35a285fc9170459395ef7304b4c468d6', 'request_id': 'req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:03.320566 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '35a285fc9170459395ef7304b4c468d6', 'request_id': 'req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:03.331261 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4 admin admin] The token cache is being invalidate because project 35a285fc9170459395ef7304b4c468d6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:09:03.331543 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:03.332174 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:03.333644 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1254/2549] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:09:03 2022] DELETE /identity/v3/projects/35a285fc9170459395ef7304b4c468d6 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:09:03.337591 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44040dd7-57ea-4e78-b8ab-a40e40b0a446 service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:03.347770 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44040dd7-57ea-4e78-b8ab-a40e40b0a446 service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:03.366581 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:03.408516 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a07eb9-0ab0-4c4e-8f41-4aec1132e7df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:03.409158 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a07eb9-0ab0-4c4e-8f41-4aec1132e7df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:03.409443 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a07eb9-0ab0-4c4e-8f41-4aec1132e7df admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:03.409719 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a07eb9-0ab0-4c4e-8f41-4aec1132e7df admin admin] PATH_INFO: `/v3/users/09f09b11a81842308dde5186f5754138` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:03.410931 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a07eb9-0ab0-4c4e-8f41-4aec1132e7df admin admin] RBAC: Authorizing `identity:delete_user(user_id=09f09b11a81842308dde5186f5754138)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:03.412909 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a07eb9-0ab0-4c4e-8f41-4aec1132e7df admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:03.459809 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c7a07eb9-0ab0-4c4e-8f41-4aec1132e7df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '09f09b11a81842308dde5186f5754138'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:03.460454 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c7a07eb9-0ab0-4c4e-8f41-4aec1132e7df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '09f09b11a81842308dde5186f5754138'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:03.468218 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c7a07eb9-0ab0-4c4e-8f41-4aec1132e7df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '09f09b11a81842308dde5186f5754138'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:03.480449 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c7a07eb9-0ab0-4c4e-8f41-4aec1132e7df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '09f09b11a81842308dde5186f5754138'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:03.487343 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c7a07eb9-0ab0-4c4e-8f41-4aec1132e7df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '09f09b11a81842308dde5186f5754138'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:03.493220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1296/2550] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:09:03 2022] DELETE /identity/v3/users/09f09b11a81842308dde5186f5754138 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:09:03.765257 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:03.766746 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:03.780731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a7f120-6fdc-410f-9106-5fbd4c4a6cc4 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:03.844345 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0b75b2-6147-4d19-862a-f0e52827c6b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:03.844983 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0b75b2-6147-4d19-862a-f0e52827c6b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:03.845265 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0b75b2-6147-4d19-862a-f0e52827c6b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:03.845542 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0b75b2-6147-4d19-862a-f0e52827c6b4 admin admin] PATH_INFO: `/v3/projects/413fd1eccc34426cbf4341a2f0faa239` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:03.846663 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0b75b2-6147-4d19-862a-f0e52827c6b4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=413fd1eccc34426cbf4341a2f0faa239)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:03.848622 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0b75b2-6147-4d19-862a-f0e52827c6b4 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:03.879015 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:03.897126 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8a0b75b2-6147-4d19-862a-f0e52827c6b4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '413fd1eccc34426cbf4341a2f0faa239', 'request_id': 'req-8a0b75b2-6147-4d19-862a-f0e52827c6b4'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:03.905122 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8a0b75b2-6147-4d19-862a-f0e52827c6b4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '413fd1eccc34426cbf4341a2f0faa239', 'request_id': 'req-8a0b75b2-6147-4d19-862a-f0e52827c6b4'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:03.911536 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8a0b75b2-6147-4d19-862a-f0e52827c6b4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '413fd1eccc34426cbf4341a2f0faa239', 'request_id': 'req-8a0b75b2-6147-4d19-862a-f0e52827c6b4'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:03.922264 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8a0b75b2-6147-4d19-862a-f0e52827c6b4 admin admin] The token cache is being invalidate because project 413fd1eccc34426cbf4341a2f0faa239 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:09:03.922555 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8a0b75b2-6147-4d19-862a-f0e52827c6b4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:03.923240 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-8a0b75b2-6147-4d19-862a-f0e52827c6b4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:03.924728 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1255/2551] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:09:03 2022] DELETE /identity/v3/projects/413fd1eccc34426cbf4341a2f0faa239 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:09:03.929708 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa596585-ab18-4bbc-a1e5-46f37678c257 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:03.930220 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa596585-ab18-4bbc-a1e5-46f37678c257 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:03.930337 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa596585-ab18-4bbc-a1e5-46f37678c257 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:03.951256 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-fa596585-ab18-4bbc-a1e5-46f37678c257 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:09:03.952471 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-fa596585-ab18-4bbc-a1e5-46f37678c257 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:04.011524 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1297/2552] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:09:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:09:04.019853 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171e1695-c9be-4cbe-b48c-324c8bad7526 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:04.020144 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171e1695-c9be-4cbe-b48c-324c8bad7526 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:04.020420 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171e1695-c9be-4cbe-b48c-324c8bad7526 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:04.040425 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-171e1695-c9be-4cbe-b48c-324c8bad7526 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:09:04.041643 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-171e1695-c9be-4cbe-b48c-324c8bad7526 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:04.049894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1256/2553] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:09:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:09:04.054273 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa596585-ab18-4bbc-a1e5-46f37678c257 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:04.064435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa596585-ab18-4bbc-a1e5-46f37678c257 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:04.069296 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4a3365-b40a-43a8-8f16-326491617e30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:04.069908 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4a3365-b40a-43a8-8f16-326491617e30 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:04.070194 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4a3365-b40a-43a8-8f16-326491617e30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:04.070474 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4a3365-b40a-43a8-8f16-326491617e30 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:04.071183 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4a3365-b40a-43a8-8f16-326491617e30 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:09:04.071469 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4a3365-b40a-43a8-8f16-326491617e30 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:04.073078 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4a3365-b40a-43a8-8f16-326491617e30 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:04.085314 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1298/2554] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:09:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 16 01:09:04.089593 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171e1695-c9be-4cbe-b48c-324c8bad7526 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:04.098798 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171e1695-c9be-4cbe-b48c-324c8bad7526 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:04.103556 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c308b0-7a96-40e3-8e64-9286cf0d2a20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:04.104154 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c308b0-7a96-40e3-8e64-9286cf0d2a20 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:04.104435 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c308b0-7a96-40e3-8e64-9286cf0d2a20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:04.104711 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c308b0-7a96-40e3-8e64-9286cf0d2a20 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:04.105341 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c308b0-7a96-40e3-8e64-9286cf0d2a20 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:04.107550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c308b0-7a96-40e3-8e64-9286cf0d2a20 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:04.135194 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1257/2555] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:09:04 2022] POST /identity/v3/projects => generated 381 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:09:04.140174 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4a3365-b40a-43a8-8f16-326491617e30 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:04.150288 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4a3365-b40a-43a8-8f16-326491617e30 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:04.175970 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:04.210562 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfe9b23-610d-4a19-8f0d-f2a0046fa7e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:04.211225 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfe9b23-610d-4a19-8f0d-f2a0046fa7e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:04.211513 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfe9b23-610d-4a19-8f0d-f2a0046fa7e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:04.211791 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfe9b23-610d-4a19-8f0d-f2a0046fa7e4 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:04.212435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfe9b23-610d-4a19-8f0d-f2a0046fa7e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:04.214377 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfe9b23-610d-4a19-8f0d-f2a0046fa7e4 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:04.239723 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1299/2556] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:09:04 2022] POST /identity/v3/users => generated 291 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:09:04.240895 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c308b0-7a96-40e3-8e64-9286cf0d2a20 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:04.251889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c308b0-7a96-40e3-8e64-9286cf0d2a20 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:04.256676 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd40464-78eb-4469-bd30-2d01142cd74c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:04.257303 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd40464-78eb-4469-bd30-2d01142cd74c admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:04.257585 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd40464-78eb-4469-bd30-2d01142cd74c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:04.257860 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd40464-78eb-4469-bd30-2d01142cd74c admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:04.258516 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd40464-78eb-4469-bd30-2d01142cd74c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:04.260520 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd40464-78eb-4469-bd30-2d01142cd74c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:04.272312 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1258/2557] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:09:04 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 16 01:09:04.276303 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfe9b23-610d-4a19-8f0d-f2a0046fa7e4 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:04.286455 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfe9b23-610d-4a19-8f0d-f2a0046fa7e4 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:04.291255 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81165239-c325-4fec-a6ae-6823aee76b59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:04.291858 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81165239-c325-4fec-a6ae-6823aee76b59 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:04.292144 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81165239-c325-4fec-a6ae-6823aee76b59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:04.292425 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81165239-c325-4fec-a6ae-6823aee76b59 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:04.293718 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81165239-c325-4fec-a6ae-6823aee76b59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:04.295379 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81165239-c325-4fec-a6ae-6823aee76b59 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:04.306106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1300/2558] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:09:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:09:04.311244 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd40464-78eb-4469-bd30-2d01142cd74c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:04.321860 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd40464-78eb-4469-bd30-2d01142cd74c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:04.326990 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df74e72d-d917-4946-b593-8871cbc90c43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:04.327645 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df74e72d-d917-4946-b593-8871cbc90c43 admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:04.327924 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df74e72d-d917-4946-b593-8871cbc90c43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:04.328198 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df74e72d-d917-4946-b593-8871cbc90c43 admin admin] PATH_INFO: `/v3/projects/3c5315d8d81a4a5387dbe596f2c9178a/users/7567fa69b3dc469195cd602d67d8e253/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:04.342940 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df74e72d-d917-4946-b593-8871cbc90c43 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3c5315d8d81a4a5387dbe596f2c9178a, user_id=7567fa69b3dc469195cd602d67d8e253, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:04.345621 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df74e72d-d917-4946-b593-8871cbc90c43 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:04.359009 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1259/2559] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:09:04 2022] PUT /identity/v3/projects/3c5315d8d81a4a5387dbe596f2c9178a/users/7567fa69b3dc469195cd602d67d8e253/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:09:04.370338 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9d53a6-9119-4020-a939-052819eed098 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:04.370650 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9d53a6-9119-4020-a939-052819eed098 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:04.370953 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9d53a6-9119-4020-a939-052819eed098 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:04.398982 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-5b9d53a6-9119-4020-a939-052819eed098 None None] MFA Rules not processed for user `7567fa69b3dc469195cd602d67d8e253`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:09:04.400189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-5b9d53a6-9119-4020-a939-052819eed098 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:04.465863 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1301/2560] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:09:04 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:09:04.470582 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df74e72d-d917-4946-b593-8871cbc90c43 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:04.532431 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df74e72d-d917-4946-b593-8871cbc90c43 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:04.537055 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007851fb-e81c-4614-892e-5eea43443db0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:04.537659 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007851fb-e81c-4614-892e-5eea43443db0 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:04.537945 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007851fb-e81c-4614-892e-5eea43443db0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:04.538219 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007851fb-e81c-4614-892e-5eea43443db0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:04.538844 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007851fb-e81c-4614-892e-5eea43443db0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:04.540817 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007851fb-e81c-4614-892e-5eea43443db0 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:04.567940 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1260/2561] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:09:04 2022] POST /identity/v3/projects => generated 383 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:09:04.572035 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9d53a6-9119-4020-a939-052819eed098 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:04.583141 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9d53a6-9119-4020-a939-052819eed098 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:04.642710 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb418768-2e79-4398-b1ba-a5d7e3c3ed92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:04.643365 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb418768-2e79-4398-b1ba-a5d7e3c3ed92 admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:04.643651 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb418768-2e79-4398-b1ba-a5d7e3c3ed92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:04.643924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb418768-2e79-4398-b1ba-a5d7e3c3ed92 admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:04.644553 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb418768-2e79-4398-b1ba-a5d7e3c3ed92 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:04.646498 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb418768-2e79-4398-b1ba-a5d7e3c3ed92 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:04.669224 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1302/2562] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:09:04 2022] POST /identity/v3/users => generated 298 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:09:04.673510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007851fb-e81c-4614-892e-5eea43443db0 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:04.684213 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007851fb-e81c-4614-892e-5eea43443db0 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:04.688902 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a21a972-a7c2-48a2-a6fd-aaf95463662a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:04.689495 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a21a972-a7c2-48a2-a6fd-aaf95463662a admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:04.689777 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a21a972-a7c2-48a2-a6fd-aaf95463662a admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:04.690051 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a21a972-a7c2-48a2-a6fd-aaf95463662a admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:04.690719 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a21a972-a7c2-48a2-a6fd-aaf95463662a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:04.693002 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a21a972-a7c2-48a2-a6fd-aaf95463662a admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:04.704339 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1261/2563] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:09:04 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:09:04.708738 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb418768-2e79-4398-b1ba-a5d7e3c3ed92 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:04.719717 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb418768-2e79-4398-b1ba-a5d7e3c3ed92 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:04.724404 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdf905e-2f45-465a-82bc-9da91f4e05c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:04.725112 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdf905e-2f45-465a-82bc-9da91f4e05c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:04.725397 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdf905e-2f45-465a-82bc-9da91f4e05c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:04.725675 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdf905e-2f45-465a-82bc-9da91f4e05c6 admin admin] PATH_INFO: `/v3/projects/a25b795de8cd43e4a9fdfd31dd3891ff/users/d839665c79b3493fb410ede4bf71b84f/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:04.741733 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdf905e-2f45-465a-82bc-9da91f4e05c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a25b795de8cd43e4a9fdfd31dd3891ff, user_id=d839665c79b3493fb410ede4bf71b84f, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:04.744439 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdf905e-2f45-465a-82bc-9da91f4e05c6 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:04.757106 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1303/2564] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:09:04 2022] PUT /identity/v3/projects/a25b795de8cd43e4a9fdfd31dd3891ff/users/d839665c79b3493fb410ede4bf71b84f/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:09:04.761109 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a21a972-a7c2-48a2-a6fd-aaf95463662a admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:04.771726 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a21a972-a7c2-48a2-a6fd-aaf95463662a admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:04.793027 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:04.833096 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53729bf5-8efb-4aae-ace3-f0b06c857257 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:04.833715 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53729bf5-8efb-4aae-ace3-f0b06c857257 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:04.834000 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53729bf5-8efb-4aae-ace3-f0b06c857257 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:04.834277 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53729bf5-8efb-4aae-ace3-f0b06c857257 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:04.834951 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53729bf5-8efb-4aae-ace3-f0b06c857257 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:04.836562 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53729bf5-8efb-4aae-ace3-f0b06c857257 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:04.847187 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1262/2565] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:09:04 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:09:04.851536 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdf905e-2f45-465a-82bc-9da91f4e05c6 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:04.857598 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:04.864385 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdf905e-2f45-465a-82bc-9da91f4e05c6 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:04.869375 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311e5eee-b0bc-4617-9470-1180e0fb7b41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:04.870037 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311e5eee-b0bc-4617-9470-1180e0fb7b41 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:04.870323 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311e5eee-b0bc-4617-9470-1180e0fb7b41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:04.870623 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311e5eee-b0bc-4617-9470-1180e0fb7b41 admin admin] PATH_INFO: `/v3/projects/a25b795de8cd43e4a9fdfd31dd3891ff/users/d839665c79b3493fb410ede4bf71b84f/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:04.872630 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311e5eee-b0bc-4617-9470-1180e0fb7b41 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a25b795de8cd43e4a9fdfd31dd3891ff, user_id=d839665c79b3493fb410ede4bf71b84f, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:04.875395 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311e5eee-b0bc-4617-9470-1180e0fb7b41 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:04.885824 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1304/2566] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:09:04 2022] PUT /identity/v3/projects/a25b795de8cd43e4a9fdfd31dd3891ff/users/d839665c79b3493fb410ede4bf71b84f/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:09:04.896569 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf30804-26c2-4eaf-92f7-d1d6b0120d72 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:04.896856 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf30804-26c2-4eaf-92f7-d1d6b0120d72 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:04.897132 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf30804-26c2-4eaf-92f7-d1d6b0120d72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:04.926062 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-2bf30804-26c2-4eaf-92f7-d1d6b0120d72 None None] MFA Rules not processed for user `d839665c79b3493fb410ede4bf71b84f`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:09:04.927485 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-2bf30804-26c2-4eaf-92f7-d1d6b0120d72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:04.995647 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1263/2567] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:09:04 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:09:05.007804 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311e5eee-b0bc-4617-9470-1180e0fb7b41 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:05.009979 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-311e5eee-b0bc-4617-9470-1180e0fb7b41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:05.068331 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311e5eee-b0bc-4617-9470-1180e0fb7b41 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:05.071993 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1419f06c-9125-4d9b-843e-8ff0b9b281f4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:05.072607 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1419f06c-9125-4d9b-843e-8ff0b9b281f4 service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:05.072889 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1419f06c-9125-4d9b-843e-8ff0b9b281f4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:05.073168 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1419f06c-9125-4d9b-843e-8ff0b9b281f4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:05.082722 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1419f06c-9125-4d9b-843e-8ff0b9b281f4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:05.084239 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1419f06c-9125-4d9b-843e-8ff0b9b281f4 service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:05.094585 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1305/2568] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:09:05 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:09:05.167260 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf30804-26c2-4eaf-92f7-d1d6b0120d72 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:05.177182 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf30804-26c2-4eaf-92f7-d1d6b0120d72 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:05.180942 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5e4d1b-6f0e-4edc-aafa-e82b651871f7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:05.181566 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5e4d1b-6f0e-4edc-aafa-e82b651871f7 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:05.181847 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5e4d1b-6f0e-4edc-aafa-e82b651871f7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:05.182121 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5e4d1b-6f0e-4edc-aafa-e82b651871f7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:05.242152 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5e4d1b-6f0e-4edc-aafa-e82b651871f7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:05.243627 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5e4d1b-6f0e-4edc-aafa-e82b651871f7 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:05.253721 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1264/2569] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:09:05 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:09:05.516482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1419f06c-9125-4d9b-843e-8ff0b9b281f4 service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:05.579882 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1419f06c-9125-4d9b-843e-8ff0b9b281f4 service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:05.584336 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811a2237-80a0-4840-89bf-b7440366c679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:05.584965 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811a2237-80a0-4840-89bf-b7440366c679 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:05.585246 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811a2237-80a0-4840-89bf-b7440366c679 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:05.585520 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811a2237-80a0-4840-89bf-b7440366c679 admin admin] PATH_INFO: `/v3/users/7567fa69b3dc469195cd602d67d8e253` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:05.586627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811a2237-80a0-4840-89bf-b7440366c679 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7567fa69b3dc469195cd602d67d8e253)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:05.588588 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811a2237-80a0-4840-89bf-b7440366c679 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:05.635342 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-811a2237-80a0-4840-89bf-b7440366c679 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7567fa69b3dc469195cd602d67d8e253'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:05.635997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-811a2237-80a0-4840-89bf-b7440366c679 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7567fa69b3dc469195cd602d67d8e253'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:05.644131 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-811a2237-80a0-4840-89bf-b7440366c679 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7567fa69b3dc469195cd602d67d8e253'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:05.656379 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-811a2237-80a0-4840-89bf-b7440366c679 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7567fa69b3dc469195cd602d67d8e253'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:05.663240 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-811a2237-80a0-4840-89bf-b7440366c679 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7567fa69b3dc469195cd602d67d8e253'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:05.669958 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1306/2570] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:09:05 2022] DELETE /identity/v3/users/7567fa69b3dc469195cd602d67d8e253 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:09:05.674425 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e60d6d-2391-488c-8c9b-60ac8c4d3a3e None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:05.674714 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e60d6d-2391-488c-8c9b-60ac8c4d3a3e None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:05.675027 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e60d6d-2391-488c-8c9b-60ac8c4d3a3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:05.694792 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-b8e60d6d-2391-488c-8c9b-60ac8c4d3a3e None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:09:05.696022 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b8e60d6d-2391-488c-8c9b-60ac8c4d3a3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:05.735638 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=100 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:05.756194 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1265/2571] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:09:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:09:05.765972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811a2237-80a0-4840-89bf-b7440366c679 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:05.767398 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-811a2237-80a0-4840-89bf-b7440366c679 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:05.780997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:05.827257 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811a2237-80a0-4840-89bf-b7440366c679 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:05.831077 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7dbd67-c058-4f6e-99ae-ce89b67b7f34 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:05.831656 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7dbd67-c058-4f6e-99ae-ce89b67b7f34 service neutron] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:05.831941 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7dbd67-c058-4f6e-99ae-ce89b67b7f34 service neutron] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:05.832218 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7dbd67-c058-4f6e-99ae-ce89b67b7f34 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:05.841298 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7dbd67-c058-4f6e-99ae-ce89b67b7f34 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:05.842744 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7dbd67-c058-4f6e-99ae-ce89b67b7f34 service neutron] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:05.852362 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1307/2572] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:09:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:09:06.125295 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e60d6d-2391-488c-8c9b-60ac8c4d3a3e None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:06.126949 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b8e60d6d-2391-488c-8c9b-60ac8c4d3a3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:06.142881 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e60d6d-2391-488c-8c9b-60ac8c4d3a3e None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:06.148863 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2da425f-501a-4623-9c10-91043132f7e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:06.149519 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2da425f-501a-4623-9c10-91043132f7e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:06.149860 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2da425f-501a-4623-9c10-91043132f7e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:06.150057 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2da425f-501a-4623-9c10-91043132f7e1 admin admin] PATH_INFO: `/v3/projects/3c5315d8d81a4a5387dbe596f2c9178a` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:06.151271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2da425f-501a-4623-9c10-91043132f7e1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3c5315d8d81a4a5387dbe596f2c9178a)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:06.153348 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2da425f-501a-4623-9c10-91043132f7e1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:06.202242 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e2da425f-501a-4623-9c10-91043132f7e1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3c5315d8d81a4a5387dbe596f2c9178a', 'request_id': 'req-e2da425f-501a-4623-9c10-91043132f7e1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:06.211300 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e2da425f-501a-4623-9c10-91043132f7e1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3c5315d8d81a4a5387dbe596f2c9178a', 'request_id': 'req-e2da425f-501a-4623-9c10-91043132f7e1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:06.219353 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e2da425f-501a-4623-9c10-91043132f7e1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3c5315d8d81a4a5387dbe596f2c9178a', 'request_id': 'req-e2da425f-501a-4623-9c10-91043132f7e1'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:06.230007 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e2da425f-501a-4623-9c10-91043132f7e1 admin admin] The token cache is being invalidate because project 3c5315d8d81a4a5387dbe596f2c9178a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:09:06.230312 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e2da425f-501a-4623-9c10-91043132f7e1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:06.231563 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-e2da425f-501a-4623-9c10-91043132f7e1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:06.233043 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1266/2573] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:09:06 2022] DELETE /identity/v3/projects/3c5315d8d81a4a5387dbe596f2c9178a => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:09:06.237029 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7dbd67-c058-4f6e-99ae-ce89b67b7f34 service neutron] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:06.246564 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7dbd67-c058-4f6e-99ae-ce89b67b7f34 service neutron] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:06.305040 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa57acc-f3fd-4184-9afa-5733acf00f1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:06.305686 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa57acc-f3fd-4184-9afa-5733acf00f1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:06.305972 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa57acc-f3fd-4184-9afa-5733acf00f1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:06.306250 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa57acc-f3fd-4184-9afa-5733acf00f1e admin admin] PATH_INFO: `/v3/users/d839665c79b3493fb410ede4bf71b84f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:06.307408 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa57acc-f3fd-4184-9afa-5733acf00f1e admin admin] RBAC: Authorizing `identity:delete_user(user_id=d839665c79b3493fb410ede4bf71b84f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:06.309438 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa57acc-f3fd-4184-9afa-5733acf00f1e admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:06.354769 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-8aa57acc-f3fd-4184-9afa-5733acf00f1e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd839665c79b3493fb410ede4bf71b84f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:06.355948 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-8aa57acc-f3fd-4184-9afa-5733acf00f1e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd839665c79b3493fb410ede4bf71b84f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:06.363950 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-8aa57acc-f3fd-4184-9afa-5733acf00f1e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd839665c79b3493fb410ede4bf71b84f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:06.376163 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-8aa57acc-f3fd-4184-9afa-5733acf00f1e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd839665c79b3493fb410ede4bf71b84f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:06.382994 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-8aa57acc-f3fd-4184-9afa-5733acf00f1e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd839665c79b3493fb410ede4bf71b84f'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:06.389538 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1308/2574] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:09:06 2022] DELETE /identity/v3/users/d839665c79b3493fb410ede4bf71b84f => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:09:06.655938 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2da425f-501a-4623-9c10-91043132f7e1 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:06.657601 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-e2da425f-501a-4623-9c10-91043132f7e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:06.671550 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2da425f-501a-4623-9c10-91043132f7e1 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:06.730004 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63978e19-c0ae-4fd2-bc7c-5f998d1428c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:06.731271 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63978e19-c0ae-4fd2-bc7c-5f998d1428c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:06.731555 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63978e19-c0ae-4fd2-bc7c-5f998d1428c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:06.731846 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63978e19-c0ae-4fd2-bc7c-5f998d1428c6 admin admin] PATH_INFO: `/v3/projects/a25b795de8cd43e4a9fdfd31dd3891ff` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:06.732957 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63978e19-c0ae-4fd2-bc7c-5f998d1428c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a25b795de8cd43e4a9fdfd31dd3891ff)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:06.734941 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63978e19-c0ae-4fd2-bc7c-5f998d1428c6 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:06.781452 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-63978e19-c0ae-4fd2-bc7c-5f998d1428c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a25b795de8cd43e4a9fdfd31dd3891ff', 'request_id': 'req-63978e19-c0ae-4fd2-bc7c-5f998d1428c6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:06.788429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-63978e19-c0ae-4fd2-bc7c-5f998d1428c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a25b795de8cd43e4a9fdfd31dd3891ff', 'request_id': 'req-63978e19-c0ae-4fd2-bc7c-5f998d1428c6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:06.795431 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-63978e19-c0ae-4fd2-bc7c-5f998d1428c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a25b795de8cd43e4a9fdfd31dd3891ff', 'request_id': 'req-63978e19-c0ae-4fd2-bc7c-5f998d1428c6'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:06.801795 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:06.807213 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-63978e19-c0ae-4fd2-bc7c-5f998d1428c6 admin admin] The token cache is being invalidate because project a25b795de8cd43e4a9fdfd31dd3891ff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81485) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:09:06.807708 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-63978e19-c0ae-4fd2-bc7c-5f998d1428c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:06.808780 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-63978e19-c0ae-4fd2-bc7c-5f998d1428c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:06.810714 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1267/2575] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:09:06 2022] DELETE /identity/v3/projects/a25b795de8cd43e4a9fdfd31dd3891ff => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:09:06.815602 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f76b9be-d96e-48f5-b095-81ef094d4c1a None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:06.815896 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f76b9be-d96e-48f5-b095-81ef094d4c1a None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:06.816216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f76b9be-d96e-48f5-b095-81ef094d4c1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:06.838920 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-6f76b9be-d96e-48f5-b095-81ef094d4c1a None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:09:06.840933 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-6f76b9be-d96e-48f5-b095-81ef094d4c1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:06.871243 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:06.904240 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1309/2576] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:09:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:09:06.911987 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740b7e57-71cd-41e9-9c4b-942c20a87ec7 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:06.912273 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740b7e57-71cd-41e9-9c4b-942c20a87ec7 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:06.912547 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740b7e57-71cd-41e9-9c4b-942c20a87ec7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:06.932884 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-740b7e57-71cd-41e9-9c4b-942c20a87ec7 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:09:06.934178 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-740b7e57-71cd-41e9-9c4b-942c20a87ec7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:06.942055 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1268/2577] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:09:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:09:06.946646 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f76b9be-d96e-48f5-b095-81ef094d4c1a None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:06.956502 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f76b9be-d96e-48f5-b095-81ef094d4c1a None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:06.961907 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4378f508-bda1-44bc-858a-b3cb3e165933 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:06.962583 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4378f508-bda1-44bc-858a-b3cb3e165933 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:06.962889 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4378f508-bda1-44bc-858a-b3cb3e165933 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:06.963203 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4378f508-bda1-44bc-858a-b3cb3e165933 admin admin] PATH_INFO: `/v3/domains` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:06.963857 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4378f508-bda1-44bc-858a-b3cb3e165933 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81486) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 16 01:09:06.964167 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4378f508-bda1-44bc-858a-b3cb3e165933 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:06.966562 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4378f508-bda1-44bc-858a-b3cb3e165933 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:06.978340 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1310/2578] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 16 01:09:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 16 01:09:06.983486 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740b7e57-71cd-41e9-9c4b-942c20a87ec7 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:06.992672 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740b7e57-71cd-41e9-9c4b-942c20a87ec7 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:06.997401 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0ae924-07fa-465a-9db2-faf21912afa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:06.997995 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0ae924-07fa-465a-9db2-faf21912afa1 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:06.998273 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0ae924-07fa-465a-9db2-faf21912afa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:06.998541 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0ae924-07fa-465a-9db2-faf21912afa1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:06.999207 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0ae924-07fa-465a-9db2-faf21912afa1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:07.001104 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0ae924-07fa-465a-9db2-faf21912afa1 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:07.029060 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1269/2579] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:09:06 2022] POST /identity/v3/projects => generated 385 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:09:07.033482 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4378f508-bda1-44bc-858a-b3cb3e165933 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:07.044307 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4378f508-bda1-44bc-858a-b3cb3e165933 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:07.100173 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43af3d6f-7c8b-4c81-8181-87425bc23a4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:07.100792 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43af3d6f-7c8b-4c81-8181-87425bc23a4d admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:07.101078 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43af3d6f-7c8b-4c81-8181-87425bc23a4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:07.101357 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43af3d6f-7c8b-4c81-8181-87425bc23a4d admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:07.101990 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43af3d6f-7c8b-4c81-8181-87425bc23a4d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:07.103964 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43af3d6f-7c8b-4c81-8181-87425bc23a4d admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:07.125633 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1311/2580] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:09:07 2022] POST /identity/v3/users => generated 293 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:09:07.129788 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0ae924-07fa-465a-9db2-faf21912afa1 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:07.141141 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0ae924-07fa-465a-9db2-faf21912afa1 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:07.145932 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7d4c69-5ea3-4502-bc67-5048cdbe9347 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:07.146532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7d4c69-5ea3-4502-bc67-5048cdbe9347 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:07.146815 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7d4c69-5ea3-4502-bc67-5048cdbe9347 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:07.147118 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7d4c69-5ea3-4502-bc67-5048cdbe9347 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:07.147763 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7d4c69-5ea3-4502-bc67-5048cdbe9347 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:07.149368 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7d4c69-5ea3-4502-bc67-5048cdbe9347 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:07.160532 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1270/2581] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:09:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:09:07.164570 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43af3d6f-7c8b-4c81-8181-87425bc23a4d admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:07.174782 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43af3d6f-7c8b-4c81-8181-87425bc23a4d admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:07.179637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ddc609-6ead-4be8-8065-04dca737a9e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:07.180254 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ddc609-6ead-4be8-8065-04dca737a9e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:07.180539 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ddc609-6ead-4be8-8065-04dca737a9e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:07.180819 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ddc609-6ead-4be8-8065-04dca737a9e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:07.181471 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ddc609-6ead-4be8-8065-04dca737a9e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:07.183189 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ddc609-6ead-4be8-8065-04dca737a9e6 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:07.195682 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1312/2582] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:09:07 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:09:07.200673 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7d4c69-5ea3-4502-bc67-5048cdbe9347 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:07.211039 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7d4c69-5ea3-4502-bc67-5048cdbe9347 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:07.216567 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57ecbd9-05a4-4045-ad6a-9067ebf45dcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:07.217219 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57ecbd9-05a4-4045-ad6a-9067ebf45dcc admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:07.217500 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57ecbd9-05a4-4045-ad6a-9067ebf45dcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:07.217775 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57ecbd9-05a4-4045-ad6a-9067ebf45dcc admin admin] PATH_INFO: `/v3/projects/fadabd77b39049ce87be1b401cb1e77f/users/5177732e9bf942e1af3fa36ee5256fae/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:07.232510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57ecbd9-05a4-4045-ad6a-9067ebf45dcc admin admin] RBAC: Authorizing `identity:create_grant(project_id=fadabd77b39049ce87be1b401cb1e77f, user_id=5177732e9bf942e1af3fa36ee5256fae, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:07.235223 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57ecbd9-05a4-4045-ad6a-9067ebf45dcc admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:07.248352 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1271/2583] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:09:07 2022] PUT /identity/v3/projects/fadabd77b39049ce87be1b401cb1e77f/users/5177732e9bf942e1af3fa36ee5256fae/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:09:07.260047 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ed1014-94c5-499f-b197-4a4cede2b04d None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:07.260572 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ed1014-94c5-499f-b197-4a4cede2b04d None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:07.260698 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ed1014-94c5-499f-b197-4a4cede2b04d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:07.289637 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-82ed1014-94c5-499f-b197-4a4cede2b04d None None] MFA Rules not processed for user `5177732e9bf942e1af3fa36ee5256fae`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:09:07.290880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-82ed1014-94c5-499f-b197-4a4cede2b04d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:07.355819 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1313/2584] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:09:07 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:09:07.360287 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57ecbd9-05a4-4045-ad6a-9067ebf45dcc admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:07.421858 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57ecbd9-05a4-4045-ad6a-9067ebf45dcc admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:07.426484 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d35219-a6cd-477d-93e2-b2090db291b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:07.427116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d35219-a6cd-477d-93e2-b2090db291b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:07.427453 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d35219-a6cd-477d-93e2-b2090db291b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:07.427669 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d35219-a6cd-477d-93e2-b2090db291b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:07.428298 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d35219-a6cd-477d-93e2-b2090db291b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:07.430221 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d35219-a6cd-477d-93e2-b2090db291b5 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:07.458048 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1272/2585] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:09:07 2022] POST /identity/v3/projects => generated 383 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:09:07.462194 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ed1014-94c5-499f-b197-4a4cede2b04d None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:07.474253 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ed1014-94c5-499f-b197-4a4cede2b04d None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:07.499939 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:07.533026 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd0e2f1-5bcc-45b9-85a4-e2182945760a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:07.533647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd0e2f1-5bcc-45b9-85a4-e2182945760a admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:07.533930 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd0e2f1-5bcc-45b9-85a4-e2182945760a admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:07.534202 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd0e2f1-5bcc-45b9-85a4-e2182945760a admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:07.534830 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd0e2f1-5bcc-45b9-85a4-e2182945760a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:07.536796 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd0e2f1-5bcc-45b9-85a4-e2182945760a admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:07.558645 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1314/2586] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:09:07 2022] POST /identity/v3/users => generated 292 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:09:07.562693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d35219-a6cd-477d-93e2-b2090db291b5 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:07.573363 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d35219-a6cd-477d-93e2-b2090db291b5 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:07.578130 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0231d4b-d27f-4a09-bf75-48499b4947aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:07.578731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0231d4b-d27f-4a09-bf75-48499b4947aa admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:07.579048 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0231d4b-d27f-4a09-bf75-48499b4947aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:07.579326 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0231d4b-d27f-4a09-bf75-48499b4947aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:07.580000 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0231d4b-d27f-4a09-bf75-48499b4947aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:07.581609 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0231d4b-d27f-4a09-bf75-48499b4947aa admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:07.592902 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1273/2587] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:09:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:09:07.596719 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd0e2f1-5bcc-45b9-85a4-e2182945760a admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:07.607454 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd0e2f1-5bcc-45b9-85a4-e2182945760a admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:07.612013 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d287ba07-d56c-46f0-bb0d-08342c73c71f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:07.612604 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d287ba07-d56c-46f0-bb0d-08342c73c71f admin admin] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:07.612882 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d287ba07-d56c-46f0-bb0d-08342c73c71f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:07.613153 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d287ba07-d56c-46f0-bb0d-08342c73c71f admin admin] PATH_INFO: `/v3/roles` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:07.613781 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d287ba07-d56c-46f0-bb0d-08342c73c71f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:07.615401 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d287ba07-d56c-46f0-bb0d-08342c73c71f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:07.625286 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1315/2588] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:09:07 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:09:07.629313 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0231d4b-d27f-4a09-bf75-48499b4947aa admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:07.639505 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0231d4b-d27f-4a09-bf75-48499b4947aa admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:07.644256 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bbfd13-57dc-414d-9773-1839b52ee4ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:07.644903 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bbfd13-57dc-414d-9773-1839b52ee4ee admin admin] REQUEST_METHOD: `PUT` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:07.645189 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bbfd13-57dc-414d-9773-1839b52ee4ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:07.645472 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bbfd13-57dc-414d-9773-1839b52ee4ee admin admin] PATH_INFO: `/v3/projects/5f176ff9cc2c4ea1994594e361801052/users/b68964afad1c4d31a0a055704bed53c8/roles/8a6c7a6c7d154f3db6d2639609d60224` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:07.661731 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bbfd13-57dc-414d-9773-1839b52ee4ee admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f176ff9cc2c4ea1994594e361801052, user_id=b68964afad1c4d31a0a055704bed53c8, role_id=8a6c7a6c7d154f3db6d2639609d60224)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:07.664511 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bbfd13-57dc-414d-9773-1839b52ee4ee admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:07.678435 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1274/2589] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:09:07 2022] PUT /identity/v3/projects/5f176ff9cc2c4ea1994594e361801052/users/b68964afad1c4d31a0a055704bed53c8/roles/8a6c7a6c7d154f3db6d2639609d60224 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:09:07.688746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3623efa-97fc-46e6-9e61-8bf88293a9f2 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:07.689032 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3623efa-97fc-46e6-9e61-8bf88293a9f2 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:07.689304 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3623efa-97fc-46e6-9e61-8bf88293a9f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:07.717666 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-c3623efa-97fc-46e6-9e61-8bf88293a9f2 None None] MFA Rules not processed for user `b68964afad1c4d31a0a055704bed53c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:09:07.718885 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c3623efa-97fc-46e6-9e61-8bf88293a9f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:07.783464 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1316/2590] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:09:07 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:09:07.788087 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bbfd13-57dc-414d-9773-1839b52ee4ee admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:07.833003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:07.850573 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bbfd13-57dc-414d-9773-1839b52ee4ee admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:07.855992 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4293e0-a24b-4511-bd14-05eb653c094c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:07.856668 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4293e0-a24b-4511-bd14-05eb653c094c admin admin] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:07.856944 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4293e0-a24b-4511-bd14-05eb653c094c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:07.857213 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4293e0-a24b-4511-bd14-05eb653c094c admin admin] PATH_INFO: `/v3/projects` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:07.857837 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4293e0-a24b-4511-bd14-05eb653c094c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:07.859801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4293e0-a24b-4511-bd14-05eb653c094c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:07.885998 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1275/2591] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 16 01:09:07 2022] POST /identity/v3/projects => generated 383 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:09:07.890719 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3623efa-97fc-46e6-9e61-8bf88293a9f2 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:07.901428 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3623efa-97fc-46e6-9e61-8bf88293a9f2 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:07.959040 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7826e01-310a-463f-bdba-5f37a69c76ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:07.959578 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7826e01-310a-463f-bdba-5f37a69c76ba admin admin] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:07.959865 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7826e01-310a-463f-bdba-5f37a69c76ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:07.960145 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7826e01-310a-463f-bdba-5f37a69c76ba admin admin] PATH_INFO: `/v3/users` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:07.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7826e01-310a-463f-bdba-5f37a69c76ba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:07.962711 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7826e01-310a-463f-bdba-5f37a69c76ba admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:07.985143 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1317/2592] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 16 01:09:07 2022] POST /identity/v3/users => generated 298 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 16 01:09:07.989276 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4293e0-a24b-4511-bd14-05eb653c094c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:07.999655 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4293e0-a24b-4511-bd14-05eb653c094c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:08.004210 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c453d03b-0eb4-4402-a3aa-e530b1a39d61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:08.004798 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c453d03b-0eb4-4402-a3aa-e530b1a39d61 admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:08.005074 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c453d03b-0eb4-4402-a3aa-e530b1a39d61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:08.005346 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c453d03b-0eb4-4402-a3aa-e530b1a39d61 admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:08.006111 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c453d03b-0eb4-4402-a3aa-e530b1a39d61 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:08.008233 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c453d03b-0eb4-4402-a3aa-e530b1a39d61 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:08.019871 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1276/2593] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:09:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:09:08.024435 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7826e01-310a-463f-bdba-5f37a69c76ba admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:08.035156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7826e01-310a-463f-bdba-5f37a69c76ba admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:08.039804 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2567ce7e-439f-4eeb-bb1b-1a87c20dd43f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:08.040460 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2567ce7e-439f-4eeb-bb1b-1a87c20dd43f admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:08.040746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2567ce7e-439f-4eeb-bb1b-1a87c20dd43f admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:08.041024 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2567ce7e-439f-4eeb-bb1b-1a87c20dd43f admin admin] PATH_INFO: `/v3/projects/6e0679096c8d46e697ed6604c2ae793a/users/5ba5355c3702478eb5ecd0e85b2dc50f/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:08.055890 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2567ce7e-439f-4eeb-bb1b-1a87c20dd43f admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e0679096c8d46e697ed6604c2ae793a, user_id=5ba5355c3702478eb5ecd0e85b2dc50f, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:08.058612 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2567ce7e-439f-4eeb-bb1b-1a87c20dd43f admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:08.072363 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1318/2594] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:09:08 2022] PUT /identity/v3/projects/6e0679096c8d46e697ed6604c2ae793a/users/5ba5355c3702478eb5ecd0e85b2dc50f/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:09:08.077115 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c453d03b-0eb4-4402-a3aa-e530b1a39d61 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:08.086968 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c453d03b-0eb4-4402-a3aa-e530b1a39d61 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:08.143477 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88685201-cb9b-4ef4-a741-650ea7e848ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:08.144149 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88685201-cb9b-4ef4-a741-650ea7e848ba admin admin] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:08.144350 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88685201-cb9b-4ef4-a741-650ea7e848ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:08.144619 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88685201-cb9b-4ef4-a741-650ea7e848ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:08.145251 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88685201-cb9b-4ef4-a741-650ea7e848ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:08.146835 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88685201-cb9b-4ef4-a741-650ea7e848ba admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:08.157682 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1277/2595] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 16 01:09:08 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 16 01:09:08.161880 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2567ce7e-439f-4eeb-bb1b-1a87c20dd43f admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:08.167284 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:08.173647 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2567ce7e-439f-4eeb-bb1b-1a87c20dd43f admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:08.178876 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11002bf3-4106-4086-a1f0-22edd7e98cf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:08.179605 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11002bf3-4106-4086-a1f0-22edd7e98cf9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:08.179887 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11002bf3-4106-4086-a1f0-22edd7e98cf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:08.180161 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11002bf3-4106-4086-a1f0-22edd7e98cf9 admin admin] PATH_INFO: `/v3/projects/6e0679096c8d46e697ed6604c2ae793a/users/5ba5355c3702478eb5ecd0e85b2dc50f/roles/c5cd777d70a649028f970fd986dc484f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:08.182100 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11002bf3-4106-4086-a1f0-22edd7e98cf9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e0679096c8d46e697ed6604c2ae793a, user_id=5ba5355c3702478eb5ecd0e85b2dc50f, role_id=c5cd777d70a649028f970fd986dc484f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:08.185361 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11002bf3-4106-4086-a1f0-22edd7e98cf9 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:08.196606 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1319/2596] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 16 01:09:08 2022] PUT /identity/v3/projects/6e0679096c8d46e697ed6604c2ae793a/users/5ba5355c3702478eb5ecd0e85b2dc50f/roles/c5cd777d70a649028f970fd986dc484f => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 16 01:09:08.207376 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb69dc1c-ad69-48be-8828-fa01a449f4a9 None None] REQUEST_METHOD: `POST` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:08.207693 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb69dc1c-ad69-48be-8828-fa01a449f4a9 None None] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:08.208003 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb69dc1c-ad69-48be-8828-fa01a449f4a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:08.236862 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.auth.core [None req-cb69dc1c-ad69-48be-8828-fa01a449f4a9 None None] MFA Rules not processed for user `5ba5355c3702478eb5ecd0e85b2dc50f`. Rule list: `[]` (Enabled: `True`). {{(pid=81485) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:09:08.238005 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cb69dc1c-ad69-48be-8828-fa01a449f4a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:08.306429 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1278/2597] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:09:08 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:09:08.318432 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11002bf3-4106-4086-a1f0-22edd7e98cf9 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:08.320252 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-11002bf3-4106-4086-a1f0-22edd7e98cf9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:08.377393 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11002bf3-4106-4086-a1f0-22edd7e98cf9 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:08.381074 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2650af1-76ae-475c-b83c-69d50747ac6b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:08.381682 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2650af1-76ae-475c-b83c-69d50747ac6b service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:08.381966 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2650af1-76ae-475c-b83c-69d50747ac6b service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:08.382240 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2650af1-76ae-475c-b83c-69d50747ac6b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:08.439493 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2650af1-76ae-475c-b83c-69d50747ac6b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:08.440945 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2650af1-76ae-475c-b83c-69d50747ac6b service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:08.450814 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1320/2598] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:09:08 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:09:09.961045 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb69dc1c-ad69-48be-8828-fa01a449f4a9 None None] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:09.962432 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-cb69dc1c-ad69-48be-8828-fa01a449f4a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:09.974301 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb69dc1c-ad69-48be-8828-fa01a449f4a9 None None] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:09.977836 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7479af-a8fc-4a11-990c-0dbf346c5fa8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:09.978430 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7479af-a8fc-4a11-990c-0dbf346c5fa8 service cinder] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:09.978706 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7479af-a8fc-4a11-990c-0dbf346c5fa8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:09.979015 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7479af-a8fc-4a11-990c-0dbf346c5fa8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:10.018365 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:10.037334 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7479af-a8fc-4a11-990c-0dbf346c5fa8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:10.039474 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7479af-a8fc-4a11-990c-0dbf346c5fa8 service cinder] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:10.048574 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1279/2599] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:09:09 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:09:10.587295 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2650af1-76ae-475c-b83c-69d50747ac6b service cinder] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:10.595908 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2650af1-76ae-475c-b83c-69d50747ac6b service cinder] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:10.599500 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec46908b-4c83-481b-9013-92bcdcf4e28f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '81fa2042309c48e1859b601d8bee1d5b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:10.600095 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec46908b-4c83-481b-9013-92bcdcf4e28f service cinder] REQUEST_METHOD: `GET` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:10.600370 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec46908b-4c83-481b-9013-92bcdcf4e28f service cinder] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:10.600635 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec46908b-4c83-481b-9013-92bcdcf4e28f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:10.609497 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec46908b-4c83-481b-9013-92bcdcf4e28f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:10.610943 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec46908b-4c83-481b-9013-92bcdcf4e28f service cinder] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:10.620227 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1321/2600] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:09:10 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 33 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:09:14.502153 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7479af-a8fc-4a11-990c-0dbf346c5fa8 service cinder] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:14.565249 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7479af-a8fc-4a11-990c-0dbf346c5fa8 service cinder] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:14.570023 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719b1d3b-b96e-4352-bd11-3058602dbdc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:14.570640 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719b1d3b-b96e-4352-bd11-3058602dbdc3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:14.570949 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719b1d3b-b96e-4352-bd11-3058602dbdc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:14.571225 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719b1d3b-b96e-4352-bd11-3058602dbdc3 admin admin] PATH_INFO: `/v3/users/5177732e9bf942e1af3fa36ee5256fae` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:14.572330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719b1d3b-b96e-4352-bd11-3058602dbdc3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5177732e9bf942e1af3fa36ee5256fae)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:14.574253 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719b1d3b-b96e-4352-bd11-3058602dbdc3 admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:14.621758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-719b1d3b-b96e-4352-bd11-3058602dbdc3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5177732e9bf942e1af3fa36ee5256fae'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:14.622418 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-719b1d3b-b96e-4352-bd11-3058602dbdc3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5177732e9bf942e1af3fa36ee5256fae'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:14.629732 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-719b1d3b-b96e-4352-bd11-3058602dbdc3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5177732e9bf942e1af3fa36ee5256fae'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:14.630452 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-719b1d3b-b96e-4352-bd11-3058602dbdc3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5177732e9bf942e1af3fa36ee5256fae'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:14.642999 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-719b1d3b-b96e-4352-bd11-3058602dbdc3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5177732e9bf942e1af3fa36ee5256fae'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:14.655750 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1280/2601] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:09:14 2022] DELETE /identity/v3/users/5177732e9bf942e1af3fa36ee5256fae => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:09:14.660261 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e76e82-a778-4140-9923-7b42bcceb966 None None] REQUEST_METHOD: `POST` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:14.660746 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e76e82-a778-4140-9923-7b42bcceb966 None None] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:14.660855 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e76e82-a778-4140-9923-7b42bcceb966 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:14.679997 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.auth.core [None req-a0e76e82-a778-4140-9923-7b42bcceb966 None None] MFA Rules not processed for user `68c5eeb505a240daa471059d6635deea`. Rule list: `[]` (Enabled: `True`). {{(pid=81486) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 16 01:09:14.681121 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-a0e76e82-a778-4140-9923-7b42bcceb966 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:14.741467 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1322/2602] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 16 01:09:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 16 01:09:14.751566 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719b1d3b-b96e-4352-bd11-3058602dbdc3 admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:14.752971 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-719b1d3b-b96e-4352-bd11-3058602dbdc3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:14.810939 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719b1d3b-b96e-4352-bd11-3058602dbdc3 admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:14.814611 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6985994a-feb4-4cc8-ab42-5d1c57e2a4da service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4eb1a640186411097cb1e3bb9e92020', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:14.815240 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6985994a-feb4-4cc8-ab42-5d1c57e2a4da service neutron] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:14.815522 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6985994a-feb4-4cc8-ab42-5d1c57e2a4da service neutron] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:14.815801 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6985994a-feb4-4cc8-ab42-5d1c57e2a4da service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:14.825089 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6985994a-feb4-4cc8-ab42-5d1c57e2a4da service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:14.827656 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6985994a-feb4-4cc8-ab42-5d1c57e2a4da service neutron] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:14.832594 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:14.838168 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1281/2603] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:09:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:09:15.117837 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e76e82-a778-4140-9923-7b42bcceb966 None None] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:15.119407 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-a0e76e82-a778-4140-9923-7b42bcceb966 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:15.134777 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e76e82-a778-4140-9923-7b42bcceb966 None None] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:15.140428 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90ba9b7-427c-43fa-978e-216123064925 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:15.141073 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90ba9b7-427c-43fa-978e-216123064925 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:15.142028 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90ba9b7-427c-43fa-978e-216123064925 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:15.142339 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90ba9b7-427c-43fa-978e-216123064925 admin admin] PATH_INFO: `/v3/projects/fadabd77b39049ce87be1b401cb1e77f` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:15.144199 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90ba9b7-427c-43fa-978e-216123064925 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fadabd77b39049ce87be1b401cb1e77f)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:15.146100 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90ba9b7-427c-43fa-978e-216123064925 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:15.178236 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:15.201402 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c90ba9b7-427c-43fa-978e-216123064925 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fadabd77b39049ce87be1b401cb1e77f', 'request_id': 'req-c90ba9b7-427c-43fa-978e-216123064925'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.210837 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c90ba9b7-427c-43fa-978e-216123064925 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fadabd77b39049ce87be1b401cb1e77f', 'request_id': 'req-c90ba9b7-427c-43fa-978e-216123064925'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.216985 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c90ba9b7-427c-43fa-978e-216123064925 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fadabd77b39049ce87be1b401cb1e77f', 'request_id': 'req-c90ba9b7-427c-43fa-978e-216123064925'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.228434 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c90ba9b7-427c-43fa-978e-216123064925 admin admin] The token cache is being invalidate because project fadabd77b39049ce87be1b401cb1e77f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:09:15.228810 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c90ba9b7-427c-43fa-978e-216123064925 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.229547 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-c90ba9b7-427c-43fa-978e-216123064925 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.231086 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1323/2604] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:09:15 2022] DELETE /identity/v3/projects/fadabd77b39049ce87be1b401cb1e77f => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:09:15.235325 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6985994a-feb4-4cc8-ab42-5d1c57e2a4da service neutron] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:15.245103 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6985994a-feb4-4cc8-ab42-5d1c57e2a4da service neutron] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:15.302108 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702ecebf-f172-41e7-99f4-f9b01b57b70d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:15.302889 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702ecebf-f172-41e7-99f4-f9b01b57b70d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:15.303212 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702ecebf-f172-41e7-99f4-f9b01b57b70d admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:15.303488 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702ecebf-f172-41e7-99f4-f9b01b57b70d admin admin] PATH_INFO: `/v3/users/b68964afad1c4d31a0a055704bed53c8` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:15.304615 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702ecebf-f172-41e7-99f4-f9b01b57b70d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b68964afad1c4d31a0a055704bed53c8)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:15.306546 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702ecebf-f172-41e7-99f4-f9b01b57b70d admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:15.353632 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-702ecebf-f172-41e7-99f4-f9b01b57b70d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b68964afad1c4d31a0a055704bed53c8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.354280 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-702ecebf-f172-41e7-99f4-f9b01b57b70d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b68964afad1c4d31a0a055704bed53c8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.361968 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-702ecebf-f172-41e7-99f4-f9b01b57b70d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b68964afad1c4d31a0a055704bed53c8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.362570 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-702ecebf-f172-41e7-99f4-f9b01b57b70d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b68964afad1c4d31a0a055704bed53c8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.375655 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-702ecebf-f172-41e7-99f4-f9b01b57b70d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b68964afad1c4d31a0a055704bed53c8'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.387718 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1282/2605] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:09:15 2022] DELETE /identity/v3/users/b68964afad1c4d31a0a055704bed53c8 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:09:15.654396 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90ba9b7-427c-43fa-978e-216123064925 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:15.656237 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-c90ba9b7-427c-43fa-978e-216123064925 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:15.670038 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90ba9b7-427c-43fa-978e-216123064925 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:15.727860 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:15.728500 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:15.728780 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:15.729052 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4 admin admin] PATH_INFO: `/v3/projects/5f176ff9cc2c4ea1994594e361801052` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:15.730156 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f176ff9cc2c4ea1994594e361801052)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:15.732814 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:15.779411 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f176ff9cc2c4ea1994594e361801052', 'request_id': 'req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.787061 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f176ff9cc2c4ea1994594e361801052', 'request_id': 'req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.792589 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f176ff9cc2c4ea1994594e361801052', 'request_id': 'req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.804321 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4 admin admin] The token cache is being invalidate because project 5f176ff9cc2c4ea1994594e361801052 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:09:15.804613 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.805265 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.806763 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1324/2606] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:09:15 2022] DELETE /identity/v3/projects/5f176ff9cc2c4ea1994594e361801052 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:09:15.810912 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702ecebf-f172-41e7-99f4-f9b01b57b70d admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:15.812330 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-702ecebf-f172-41e7-99f4-f9b01b57b70d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:15.825420 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702ecebf-f172-41e7-99f4-f9b01b57b70d admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:15.881901 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e2b18d-5970-43ea-82f4-0328af90869c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:15.882548 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e2b18d-5970-43ea-82f4-0328af90869c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:15.882813 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e2b18d-5970-43ea-82f4-0328af90869c admin admin] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:15.883114 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e2b18d-5970-43ea-82f4-0328af90869c admin admin] PATH_INFO: `/v3/users/5ba5355c3702478eb5ecd0e85b2dc50f` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:15.884236 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e2b18d-5970-43ea-82f4-0328af90869c admin admin] RBAC: Authorizing `identity:delete_user(user_id=5ba5355c3702478eb5ecd0e85b2dc50f)` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:15.886133 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e2b18d-5970-43ea-82f4-0328af90869c admin admin] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:15.912068 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:15.938412 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-01e2b18d-5970-43ea-82f4-0328af90869c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5ba5355c3702478eb5ecd0e85b2dc50f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.939116 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-01e2b18d-5970-43ea-82f4-0328af90869c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5ba5355c3702478eb5ecd0e85b2dc50f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.946510 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-01e2b18d-5970-43ea-82f4-0328af90869c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5ba5355c3702478eb5ecd0e85b2dc50f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.947159 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-01e2b18d-5970-43ea-82f4-0328af90869c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5ba5355c3702478eb5ecd0e85b2dc50f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.959718 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.notifications [None req-01e2b18d-5970-43ea-82f4-0328af90869c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5ba5355c3702478eb5ecd0e85b2dc50f'} {{(pid=81485) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:15.972428 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1283/2607] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 16 01:09:15 2022] DELETE /identity/v3/users/5ba5355c3702478eb5ecd0e85b2dc50f => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:09:16.253216 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4 admin admin] Authenticating user token {{(pid=81486) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:09:16.255680 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.fernet_utils [None req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81486) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:09:16.271423 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c6b3a3-51eb-4fcf-b0d0-4d2b7ca65af4 admin admin] Validating token access rules against request {{(pid=81486) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:09:16.315867 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:16.333400 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391a4624-c13d-43eb-ba5e-6695cbb45a63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '68c5eeb505a240daa471059d6635deea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f1d89d5c90cd4c35b26fbcfc82450844', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81486) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:09:16.334039 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391a4624-c13d-43eb-ba5e-6695cbb45a63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:09:16.334320 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391a4624-c13d-43eb-ba5e-6695cbb45a63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:09:16.334596 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391a4624-c13d-43eb-ba5e-6695cbb45a63 admin admin] PATH_INFO: `/v3/projects/6e0679096c8d46e697ed6604c2ae793a` {{(pid=81486) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:09:16.335750 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391a4624-c13d-43eb-ba5e-6695cbb45a63 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6e0679096c8d46e697ed6604c2ae793a)` {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:09:16.337783 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391a4624-c13d-43eb-ba5e-6695cbb45a63 admin admin] RBAC: Authorization granted {{(pid=81486) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:09:16.383175 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-391a4624-c13d-43eb-ba5e-6695cbb45a63 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6e0679096c8d46e697ed6604c2ae793a', 'request_id': 'req-391a4624-c13d-43eb-ba5e-6695cbb45a63'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:16.390924 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-391a4624-c13d-43eb-ba5e-6695cbb45a63 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6e0679096c8d46e697ed6604c2ae793a', 'request_id': 'req-391a4624-c13d-43eb-ba5e-6695cbb45a63'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:16.396627 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-391a4624-c13d-43eb-ba5e-6695cbb45a63 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6e0679096c8d46e697ed6604c2ae793a', 'request_id': 'req-391a4624-c13d-43eb-ba5e-6695cbb45a63'} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:16.409221 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-391a4624-c13d-43eb-ba5e-6695cbb45a63 admin admin] The token cache is being invalidate because project 6e0679096c8d46e697ed6604c2ae793a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81486) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 16 01:09:16.409510 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-391a4624-c13d-43eb-ba5e-6695cbb45a63 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:16.410162 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG keystone.notifications [None req-391a4624-c13d-43eb-ba5e-6695cbb45a63 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81486) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 16 01:09:16.411701 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: [pid: 81486|app: 0|req: 1325/2608] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 16 01:09:16 2022] DELETE /identity/v3/projects/6e0679096c8d46e697ed6604c2ae793a => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 16 01:09:25.968808 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:DELETE=7,keystone:SELECT=20,keystone:INSERT=1 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:09:26.406036 dv-cinder-master-3-iscsi devstack@keystone.service[81486]: DEBUG dbcounter [-] [81486] Writing DB stats keystone:SELECT=31,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81486) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 16 01:10:30.641631 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e2b18d-5970-43ea-82f4-0328af90869c admin admin] Authenticating user token {{(pid=81485) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 16 01:10:30.643083 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-01e2b18d-5970-43ea-82f4-0328af90869c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:10:30.713949 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e2b18d-5970-43ea-82f4-0328af90869c admin admin] Validating token access rules against request {{(pid=81485) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 16 01:10:30.775575 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70910cf-c6b0-473f-9e40-08a8874467df service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '87785ccd37a642e08a0ad34a4fb1c754', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '7deb2c3dae214df7895b40e7339bd6ea', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81485) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 16 01:10:30.776194 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70910cf-c6b0-473f-9e40-08a8874467df service placement] REQUEST_METHOD: `GET` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 16 01:10:30.776478 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70910cf-c6b0-473f-9e40-08a8874467df service placement] SCRIPT_NAME: `/identity` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 16 01:10:30.776758 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70910cf-c6b0-473f-9e40-08a8874467df service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81485) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 16 01:10:30.778535 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.fernet_utils [None req-b70910cf-c6b0-473f-9e40-08a8874467df service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81485) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 16 01:10:30.790583 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70910cf-c6b0-473f-9e40-08a8874467df service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 16 01:10:30.792197 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70910cf-c6b0-473f-9e40-08a8874467df service placement] RBAC: Authorization granted {{(pid=81485) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 16 01:10:30.802322 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: [pid: 81485|app: 0|req: 1284/2609] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 16 01:10:30 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 16 01:10:40.797894 dv-cinder-master-3-iscsi devstack@keystone.service[81485]: DEBUG dbcounter [-] [81485] Writing DB stats keystone:SELECT=58 {{(pid=81485) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}